ADA  154  976 


ADA  154  976 


RISK  MANAGEMENT  FOR  WEAPON  SYSTEMS  ACQUISITION:  A  DECISION  SUPPORT  SYSTEM 


Drs.  Y.  Haimes  and  V.  Chankong 
The  Systems  Engineering  Department 
Case  Western  Reserve  University 
Cleveland,  Ohio  44106 


28  February  1985 


Final  Report  for  Period  Covering  15  June  1984  -  28  February  1985 
Contract  No.  F33615-84-M-5084 


Prepared  for 

AIR  FORCE  BUSINESS  RESEARCH  MANAGEMENT  CENTER 
Wright-Patterson  AFB  Ohio  45433 


Approved  for  Public  Release: 
Unlimited  Distribution 


The  views 
and  do 


expressed  herein  are  solely  those  of  the  researcher(s) 
not  represent  those  of  the  United  States  Air  Force. 


Project  Staff 


Yacov  Y.  Haimes,  PE.,  Ph . D . 
Project  Director 


Vira  Chankong,  Ph . D . 
Project  Co-Director 


Julia  Pet-Edwards,  M.S. 
Graduate  Assistant 


C.  J.  Debeljak,  B.S.  Student 
Undergraduate  Assistant 


Colin  Clark 

Technical  Assistant,  Programmer 
Toni  Sowa 

Technical  Secretary 

Mary  Ann  Pelot 
Assistant  to  the  Director 


x 


TABLE  OF  CONTENTS 


A  \J  >  * 


Project  Staff .  i 

Table  of  Contents .  ii 

Executive  Summary .  -jv 

1.  Risk  and  Uncertainty  in  the  Weapon  Acquisition  Process— A 

Unified  Framework . . 

1.0  Introduction . ■)_! 

1.1  Uncertainties  in  the  Acquisition  Process . 1-4 

1.2  Specific  Risk  Assessment  and  Management  Needs  in  WSA: 

A  Framework . . 

2.  Risk  Assessment  and  Management  Methodology  .  2-1 

2.1  Risk  Assessment  and  Management  Process . 2-1 

2.2  What  are  Available . 2-3 

2.3  What  is  Needed:  An  Integrated  Methodological  Framework  ...  2-4 

3.  A  Decision  Support  System  for  Weapon  System  Acquisition 

Risk  Management . 3_1 

3.1  What  is  a  DSS? . 3-1 

3.2  Why  a  Decision  Support  System? . 3-2 

3.3  Components  of  a  Decision  Support  System  .  3-4 

3.4  DSS  as  a  Packaging  Tool  for  Risk  Assessment  and 

Management  in  WSA . 3-5 

3.5  Examination  of  the  Risk  Assessment  and  Management  DSS  ....  3-8 

4.  A  Prototype  DSS  for  Project  Risk  Management . 4-1 

4.1  Problems  and  Risks  in  Program  Planning  and  Management  ....  4-1 

4.2  Objectives  and  Tools  in  Project  Planning . 4-3 

4.3  Constraints  and  Complications  in  Program  Planning  .  4-5 

4.4  A  DSS  for  PM 1  s  Problems . 4-6 

4.5  Example  Problem . 4-7 

4.6  Analysis  of  Components . 4-9 

4.7  Discussion  of  Use . 4-15 

4.8  Results  and  Extensions . 4-16 

Appendix  A:  The  Acquisition  Process  .  A-l 

Appendix  B:  Risk  Assessment  and  Management  Methods:  A  Review  ....  B-l 

Appendix  C:  Causes  and  Effects  of  Risks,  Development, 

Production,  and  Deployment  Risks  .  C-l 

Appendix  D:  Details  of  Methods  in  Prototype  DSS  .  D-l 


ii 

j 


» \  V-j* 


LIST  OF  TABLES  AND  FIGURES 

Tables 

1.1  Categories  of  Uncertainty  in  the  Acquisition  Process 
Summarized  from  Discussion  of  Uncertainty  in  Lenox, 

Hamilton  T. ,  1973 .  1-12 

1.2  Typical  Decisions  made  by  the  Three  Levels  of  Authority 

in  Each  Phase  of  Acquisition  Program . 1-13 

1.3  Typical  Risk  Related  Activities  for  the  Program 

Managers  and  His  Directorates . 1-14 

3.1  Usage  Breakdown  of  Risk  Assessment  and  Management 

Tools  within  Each  Directorate . 3-1 1 

3.2  Usage  Breakdown  and  Classification  of  Tools  for  Risk 

Assessment  and  Management  within  the  Program  Office . 3-12 

4.1  Time  Estimates  for  Activities . 4-18 

4.2  Critical  Path  Analysis . . 

Figures 

1.1  Hierarchical  Networking .  x-j-j 

1.1  Simplified  View  of  Acquisition  Process  .  1-15 

1.2  Relationship  between  Uncertainties,  Risks,  and  Program 

Failure  in  Weapon  System  Acquisition  Process  .  1-16 

1.3  Levels  of  Authority  in  the  Weapon  System  Acquisition 

Process .  1_17 

1.4  Focus  of  Risk  Assessment  and  Management  Framework . 1-18 

1.5  Typical  Information  Flow  in  Risk  Assessment  and 

Management  Framework  during  Phase  III . 1_19 

2.1  A  Procedural  Framework  for  WSA  Risk  Assessment  and 

Management . 2-8 

3.1  The  "Envisioned"  DSS  for  the  Entire  Weapons  System 

Acquisition  Process . . 

4.1  Flow  Chart  of  Prototype  DSS . 4_20 

4.2  Summary  of  Example  Problem  .  4_2i 

4.3  Network  Diagram  for  Example  Problem . 4_22 

4.4  Computational  Results  for  PMRM  Analysis . 4_23 

iii 


EXECUTIVE  SUMMARY 


I.  Summary  of  Accomplishments  of  the  Feasibility  Study 

Two  principle  aims  were  set  forth  for  the  completion  study-(i)  assess 
the  feasibility  of  the  development  of  an  integrated  methodology  for  the 
overall  risk  assessment  and  management  of  weapons  system  acquisition,  and 
(ii)  assess  the  feasibility  of  translating  the  integrated  methodology  into 
a  working  tool  in  the  form  of  a  prototype  decision  support  system  (DSS). 

To  provide  the  necessary  inputs  for  the  project,  we  studied  and 
identified  specific  risk  management  issues  and  concerns  associated  with  the 
weapons  system  acquisition  process,  focusing  on: 

*  the  institutional  aspect--the  decision-making  structure  associated 
with  weapons  system  acquisition  as  well  as  the  management  instruments 
that  are  at  the  disposal  of  the  program  office. 

*  past  decision  experience--particularly  the  issues,  concerns, 
criteria,  and  premises  that  were  used  during  the  project  planning 
and  selection  state  for  certain  selected  weapons-system  acquisi¬ 
tion  projects  of  the  past  and  (based  on  past  experience) the  poten¬ 
tial  impacts  that  changes  in  the  basic  premises,  decision  environ¬ 
ment  and/or  decision  approaches  may  have  on  current  and  future 

decisions . 

*  present  and  future  decisions  that  the  USAF  and  the  program 
office  must  make,  and  the  interaction  among  decisions— past , 
present,  and  future. 

*  information  needed  to  make  decisions  and  how,  in  general,  that 
information  may  be  obtained. 

*  current  risk  management  issues,  concerns,  and  premises. 

*  potential  areas  susceptible  to  random  changes  and  uncertainties. 


From  the  above,  we  were  able  to  determine  and  accomplish  the 
following : 

*  The  major  elements  of  risk  in  the  various  phases  and  oganizat iona 1 
levels  of  the  WSA  can  be  considered  in  terms  of  two  classes-- 
internal  program  uncertainty  and  external  program  uncertainty. 
The  former  is  due  mainly  to  lack  of  information  while  the  latter 
can  be  considered  as  random  uncertainty  that  is  beyond  any  control 
from  within. 

*  Risk  of  cost  overrun,  risk  of  schedule  slippage  and  risk  of  not 
meeting  quality  requirements  are  the  major  risk  issues  in  the 
internal  program  class,  while  risk  of  budget  cut,  risk  of 
resource  scarcity,  and  risk  of  priority  change  are  of  major 
concern  in  the  external  program  class.  All  these  risks  are  to  be 
monitored  and  controlled.  Because  of  the  multiplicity  of  risk 
monitoring  and  control  objectives  in  each  class,  a  methodological 
framework  with  multiobjective  analysis  capability  was  deemed  a 
logical  choice. 

*  To  cope  with  the  inherent  complexity  and  multitude  of 

uncertainties  in  the  WSA  process,  there  is  clearly  a  need  for  an 
integrated  and  comprehensive  decision  aid  that  weaves  together  a 
number  of  carefully  selected  risk  analysis  tools  and 

methodologies.  For  this  purpose,  several  existing  risk  assessment 
and  management  methods  were  evaluated,  out  of  which  a 
potentially  useful  subset  was  chosen.  The  subset  includes  network 
modeling  and  analysis,  particularly  PERT,  VERT,  and  GERT,  the 
partitioned  multiobjective  risk  method  (PMRM),  the  surrogate  worth 
trade-off  (SWT)  method,  Bayesian  decision  analysis,  and  several 
other  risk  quantification  and  evaluation  approaches.  A 
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hierarchical,  multiobjective  risk  assessment  and  management 
framework  built  upon  the  recommended  tools  was  then  proposed  for 
the  WSA  process. 

*  To  translate  the  integrated  framework  into  a  working  tool,  we 

proposed  the  use  and  development  of  a  decision  support  system 
(DSS).  A  DSS  offers  a  great  opportunities  for  packaging  or 
modularity,  allowing  various  components  (modules)  to  be  added, 
removed,  or  set  aside  as  the  needs  arise. 

*  To  test  the  feasibility  and  effectiveness  of  the  developed 

hierarchical  multiobjective  framework,  a  specific  risk  management 
problem  (adopted  from  one  used  by  the  Defense  Systems  Management 
College  at  Fort  Belvoir,  Virginia),  was  solved.  A  prototype  DSS 
was  developed  and  used  to  demonstrate  the  proposed  concepts. 

II .  General  Re c omme nd a t i on s 

By  its  nature,  a  feasibility  study  addresses  a  broad  range  of  issues 
and  it  must  do  this  mostly  in  generalities,  necessarily  avoiding  the 
specifics.  In  the  completed  study,  we  have  identified  desirable  features, 
components,  and  characteristics  of  the  comprehensive  WSA  risk  assessment 
and  management  system,  indicated  how  these  elements  should  be  packaged  into 
one  feasible  yet  powerful  unit,  and  developed  a  prototype  DSS  to 
demonstrate  the  concepts.  With  the  ultimate  aim  of  developing  a 
comprehensive  risk  assessment  and  management  DSS  for  the  entire  weapons 
system  acquisition  process,  the  results  from  the  feasibility  study  must  be 
further  crystallized  and  consolidated  as  well  as  extended  to  include  all 
areas  of  risk  and  all  phases  of  the  process. 

Future  work  should  thus  follow  a  two-pronged  strategy,  one  involving 
crystallization  and  consolidation  and  the  other  is  more  toward  extension. 
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The  first  of  these  is  centered  around  a  packaging  concept  of  consolidating 
existing  tools  and  methodologies  into  a  working  DSS  that  is  powerful  and 
versatile  enough  to  deal  with  most  WSA  risk  issues  and  challenges  (of  any 
scope  and  level  of  complexity)  and  yet  is  easy  and  flexible  enough  to  use. 

The  second  represents  an  extension  effort  and  is  directed  more  toward 
methodological  improvements.  These  include  i)  refining  existing  risk 
quantification  and  evaluation  tools  and/or  developing  new  ones  to  better 
cater  to  the  specific  needs  of  the  WSA  process,  and  ii)  developing  or 
refining  coordination  strategies  for  coordinating  risk  assessment  and 
management  activities  at  the  various  levels  of  the  USAF  organization. 
Typical  tasks  for  the  first  type  of  study  should  include  the  following: 

A)  Expand  the  presently  developed  prototype  DSS  to  allow  the  project 
manager  (PM)  to  consider  more  general  aspects  and  attributes  in 
his  decision  process. 

B)  Develop  modules  at  the  directorate  level  that  respond  to  the 
specific  needs  that  the  directorates  have  throughout  the  program's 
life  cycle. 

Typical  tasks  for  the  second  type  of  study  are: 

C)  Explore  the  concept  of  hierarchical  network  and  decision  tree 
models  (as  illustrated  in  Figure  1.1)  and  develop  methodological 
procedures  for  them. 

D)  Develop  a  coordination  scheme  between  the  PM  and  the  directorates 
that  will  facilitate  information  flow  through  the  use  of 
hierarchical  network  and  decision  tree  models. 

III.  Proposed  Future  S tudy 

Based  on  the  general  recommendation  set  forth  in  the  previous  section, 
we  propose  to  follow-up  the  completed  feasibility  study  on  risk  management 
for  weapons  system  acquisition  with  a  two-phase  study.  Although  both 
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phases  build  on  results  derived  fro.  the  feasibility  study,  there  are 
distinctive  attributes  associated  «lth  each  phase  as  well  as  a  temporal 
logic  in  the  sequence. 

The  first  phase,  based  on  Recommendation  A,  aims  principally  at 
completing  the  development  of  a  relatively  general  but  usable  risk 
assessment  and  management  methodology  from  the  program  manager’s 
perspectives,  building  upon  the  existing  prototype  DSS  model.  The  emphasis 
here  is  placed  upon  risk  quantification  and  evaluation 

The  second  phase  is  to  carry  out  the  remaining  recommendations  1)  by 
at  both  developing  a  comprehensive  risk  assessment  and  management  DSS  for  a 
selected  program  office  and  2)  by  extending,  refining,  and  strengthening  a 
hierarchical-multiobjective  framework  and  methodology  for  the  overall  WSA 

risk  assessment  and  management  process. 

In  the  terminology  of  the  WSA  process,  we  may  view  the  just-completed 

feasibility  study  as  a  conceptual  exploration,  the  proposed  Phase  I  study 
as  a  demonstration  and  validation,  and  the  proposed  Phase  II  study 
full-scale  development. 

IV.  Phase  I  Study-Risk  Management  for  Weapons  System  Acquisition: 

A  Project  Manager’s  Perspectives 

The  overall  objective  of  Phase  I  Is  to  develop  a  usable  risk 
management  methodology  for  weapons  system  acguisition  at  the  program 
manager  level  that  is  comprehensive  in  nature  and  that  makes  explicit 
of  advanced  state-of-the-art  tools  developed  in  such  fields  as  risk 
assessment  and  management,  decision  analysis,  multiple-criteria  decision 

making,  etc.  More  specifically,  the  tasks  in  this  study  will  include  the 
following: 

_  pfrt  for  project  scheduling  with  VERT, 

a)  Replace  the  present  use  of  PERI  tor  proj 
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thus  allowing  the  use  of  probability  distrtbution  functions 
within  the  scheduling  process.  Other  network  models  will  also  be 

explored . 

b)  Quantify  the  objectives  indentified  in  the  feasibility  stud,: 
risk  of  cost  overrun 
risk  of  schedule  slippage 

risk  of  not  meeting  quality  requirements 

risk  of  budget  cost 

risk  of  resource  scarcity 

risk  of  priority  change 

Note  that  only  the  first  two  objectives  were  guantified  during  the 
feasibility  study,  and  mostly  in  a  superficial  way  (in  order  to  test  the 

overall  methodology)* 

c)  Apply  the  partitioned  multiobjective  risk  method  ( PMRM) .  This 

would  require  the  quantification  of  the  appropriate  probability 
distribution  functions  and  the  part-timing  of  the  frequency 

domain . 

d)  Apply  the  surrogate  worth  trade-off  (SWT)  method  for  the  overall 
multiobjective  risk-based  decision-making  problem.  This  would 
enable  the  project  manager  to  quantitatively  evaluate  the  various 
trade-offs  associated  with  the  various  non-commensurate 

objectives . 

e)  Demonstrate  the  working  mechanism  and  potential  usefulness  of  the 
developed  methodology  using  the  modified  and  upgraded  prototype 

DSS  . 

And,  in  anticipation  of  and  preparation  for  the  Phase  II  study,  we 
shall  also  identify  an  interested  and  willing  project  manager  within 
Wright-Patterson  Air  Force  Base  for  whom  a  specific  decision  support  system 
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will  be  developed  in  Phase  II. 

V.  Phase  II  Study -Risk  Management  for  Weapons  System  Acquisition: 

A  Comprehensive  Hierarchical-Multiobjective  Framework 

Phase  II  is  comprised  of  two  components: 

(1)  development  of  a  DSS  for  a  specific  project  manager  (in 
charge  of  weapons  systems  acquisition). 

(2)  development  of  a  more  comprehensive  risk-based  hierarchical 
multiobjective  framework  for  weapons  system  acquisition  that 
builds  on  the  results  from  the  already  completed  feasibility 
study  and  from  Phase  I 

The  anticipated  tasks  in  this  phase  can  be  found  in  Recommendations 
B }  c,  and  D,  Section  1.2.  The  following  are  some  of  our  general  thoughts 
on  those  tasks. 

The  development  of  specific  modules  for  each  directorate  should 
parallel  the  development  of  the  PM 1 s  DSS.  Once  a  complete  version  of  the 

prototype  has  been  developed  for  the  PM,  it  can  be  modified  to  respond  to 

the  more  detailed  needs  at  the  directorate  level.  Specifically,  the 
network  or  decision- tree  model  will  be  altered  to  focus  on  the  subtasks 
associated  with  each  directorate.  (For  example,  the  manufacturing 

management  directorate's  objective  is  to  assure  that  a  system  can  be 

produced  in  the  most  cost-effective  manner.  This  involves  producib ll i ty 
studies,  design  reviews,  quantification  of  production  risk,  and  production 
plans.  The  development  of  a  specific  DSS  with  modules  aimed  at  these 
duties  will  improve  the  directorate's  performance.)  Some  of  the  specific 
tools  and  extensions  that  will  be  useful  at  this  level  include. 

*  extension  of  the  work  breakdown  structure  and  method  of  moments 

*  subjective  probability  assessments 
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★  cost-estimation  techniques 

★  cross-impact  analysis 

★  probability-,  decision-,  fault-,  and  event-tree  analysis 

★  reliability  analysis 

★  network  analysis 

The  coordination  scheme,  which  will  reside  in  the  data-base  management 
system  of  the  DSS,  is  envisioned  as  some  form  of  hierarchical  networking 
scheme  that  uses  hierarchical  decision  trees  and  possibly  hierarchical 

work-breakdown  structures  (See  Figure  1.1). 

Finally,  the  development  of  the  coordination  scheme  will  allow  the 
program  office  to  take  full  advantage  of  the  micro-based  DSS  s.  By 
incorporating  existing  methodologies  and  tools  from  hierarchical 
multiobjective  coordination,  we  will  be  able  to  link  the  PM  with  his 
directorates.  This  will  provide  the  PM  with  the  specific  information  he 
needs  from  his  directorates  when  he  needs  it.  It  also  provides  the  PM  with 
an  insight  into  the  effects  of  his  decisions  on  each  of  his  directorates. 
As  for  the  directorates,  they  will  be  given  a  better  understanding  of  their 
goals  as  they  relate  to  the  overall  goals  of  the  program.  The  coordination 
scheme  will  ensure  that  each  directorate  understands  what  the  others  are 
doing  so  that  they  can  work  together  toward  the  same  goal.  It  will  help 
the  directorates  to  understand  the  actions  of  the  PM  and  also  help  them  to 
support  him. 
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Risk  and  Uncertainty  in  the  Weapon  Acquisition  Process--A  Unified 
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Framework 

1.0.  INTRODUCTION 

How  to  deal  with  the  risks  and  uncertainties  associated  with  the 
development  of  new  technologies  is  one  of  the  most  complex  and  perplexing 
problems  that  a  program  manager  must  deal  with  during  the  weapon  system 
development  and  procurement  process.  New  technology  can  range  from  minor 
modifications  of  existing  systems  to  radical,  far-reaching  innovations,  and 
may  result  in  a  new  product,  a  new  process,  or  a  combination  of  both.  Yet 
program  managers  for  the  development  of  new  technologies  must  be  able  to 
make  decisions  concerning  schedules,  budget  targets,  performance 
requirements,  and  other  aspects  of  the  program,  under  a  very  uncertain 
future . 

Because  of  the  inherent  uncertainties  associated  with  developing  and/ 
or  modifying  technologies,  significant  risks  are  involved  with  any  program. 
Such  risks  include  time  delays,  cost  overruns,  failure  to  meet  performance 
requirements  and,  in  general,  the  success  or  failure  of  the  program.  It  is 
for  this  reason  that  the  Department  of  Defense  and  Air  Force  acquisition 
management  directives  require  that  risks  be  continually  addressed 
throughout  the  development  of  major  weapon  systems. 

Currently,  no  standardized  procedures  are  in  place  to  help  program 
managers  account  for  these  risks.  Because  the  failure  of  a  major  weapons 
system  is  both  costly  and  harmful  to  national  security,  the  need  for  a 
sound  and  well-grounded  risk  assessment  and  management  methodology  must  be 
emphasized.  It  is  to  this  issue  that  we  are  addressing  ourselves. 

The  principal  aim  of  the  study  is  to  assess  the  feasibility  of  (i)  the 
development  of  an  integrated  methodology  for  the  overall  risk  assessment 
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and  management  of  weapons  systems  acquisition  (WSA),  and  (ii)  the 

translation  of  the  integrated  methodology  into  a  working  tool  in  the  form 
of  a  comprehensive  decision  support  system  (DSS) .  This  report  summarizes 
our  main  findings  and  presents  a  plan  for  future  works.  Section  1 

identifies  the  elements  of  risks  and  uncertainties  associated  with  weapons 
system  acquisition  based  on  our  view  of  the  acquisition  process  summarized 
in  Appendix  B.  We  focus  our  attention  to  the  program  office  and,  in 

particular,  the  program  manager  and  the  elements  of  risk  and  uncertainty 
that  he  must  face.  We  also  present  a  general  framework  which  can  summarize 

the  major  elements  of  the  decision-making-  process  throughout  a  program's 

activities  in  the  directorates  of  the  program  office.  The  risk  assessment 
and  management  decision  aid,  which  is  based  on  the  developed  framework  is 
an  integration  of  tools  and  concepts,  including  decision  analysis, 
hierarchical  and  and  multiobjective  decision  making,  network  analysis  and  a 
number  of  risk  assessment  and  management  methodologies.  Future  work  can  be 
directed  at  completing  the  details  of  this  framework  as  a  unified  decision 
aid  for  use  by  any  program  office  and  its  various  divisions  to  use  in 
evaluating  and  managing  the  major  areas  of  risk  during  the  entire 
acquisition  process. 

A  large  body  of  knowledge  concerning  risk  identification  and 
management  is  available.  A  review  of  this  knowledge  is  found  in  Section  2 
and  Appendix  B,  where  we  also  summarize  and  compare  a  number  of  currently 
available  methods  and  indicate  some  of  their  shortcomings  as  well  as 
provide  some  suggestions  for  combining  and  improving  them. 

In  Section  3,  we  discuss  the  useof  a  decision  support  system  (DSS)  for 
risk  management  during  the  entire  weapon  acquisiton  process.  We 
particularly  focus  on  the  production  and  deployment  stage  and  discuss 
several  selected  tools  and  concepts  which  we  combined  into  a  unified 
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package  as  a  prototype  DSS.  In  order  to  demonstrate  the  capabilities  of  our 
prototype  DSS  and  provide  a  sampling  of  its  outputs,  we  obtained  (adapted 
from  Ingalls,  1984a)  a  sample  problem  relating  to  the  production  and 
deployment  phase  of  a  project  and  applied  the  DSS  to  this  problem.  A 
detailed  description  of  the  problem  and  a  summary  of  the  results  are  also 
found  in  Section  4. 

It  is  worth  keeping  in  mind  that  the  present  project  was  a  feasibility 
study  and  consequently  the  developed  prototype  DSS  is  limited  in  both  its 
scope  and  use.  The  DSS  is  intended  to  demonstrate  the  fundamental 
strengths  and  capabilities  of  the  developed  framework.  We  are  hopeful  that 
it  provides  the  foundations  upon  which  a  more  representative  and  useable 
DSS  can  be  developed  in  the  future. 

The  success  or  failure  of  any  program  can  be  based  on  many  different 
factors.  The  entire  weapon  acquisition  process  is  tedious  and  complicated, 
consisting  of  interlocking  decisions,  many  simultaneous  activities  and 
various  planned  and  unplanned  events.  Typical  decisions  in  the  process 
involve  make-or-buy  options,  quantity,  quality  and  timing  of  products  to  be 
acquired,  selection  of  contractor,  and  management,  control  and  coordination 
of  system  development.  Within  each  of  these  decisions  lies  an  element  of 
risk  and  uncertainty.  There  is  a  need  to  account  for  these  elements  of 
risk  and  uncertainty  in  a  satisfactory  way  to  ensure  successful 

implementation  of  a  procurement  program. 

In  this  section  we  identify  the  major  areas  in  the  weapons  system 
acquisition  process  as  faced  by  the  program  office,  where  the  assessment 
and  management  of  risk  are  essential.  This  is  based  on  our  view  of  the 
process  summarized  in  Appendix  A.  We  attempt  to  identify  and  classify  the 
specific  needs  and  requirements  of  these  major  areas  as  a  means  of 
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describing  a  unified  framework  that  can  be  used  to  evaluate  and  manage  the 
major  areas  of  risk  throughout  the  acquisitions  process  by  each  component 
of  the  program  office.  The  remainder  of  this  section  focuses  on  the  risk 
activities  of  the  directorates  in  the  program  office.  We  outline  this 
phase  in  greater  depth,  to  set  the  stage  for  the  following  chapters  where  a 
prototype  decision  support  system  is  developed. 

1. 1  Uncertainties  in  the  Acquisition  Process 

Careful  examination  of  the  acquisition  process  (See  Appendix  A) 
reveals  that  the  program  office  must  continually  identify,  evaluate,  and 
manage  the  risks  and  uncertainties  during  the  entire  life  of  the  program. 
Figure  1.1  provides  a  simplified  view  of  the  acquisition  process  and 
indicates  the  specific  times  when  risk  assessment  and  management  are 
necessary . 

The  uncertainties  found  in  the  weapon  system  acquisition  process  can 
be  categorized  in  a  number  of  different  ways.  One  useful  classification 
divides  uncertainty  into  four  groups:  target,  technical,  internal  program 
and  process  uncertainties  (Lenox,  1973).  A  summary  of  this  classification 
scheme  is  given  in  Table  1.1. 

Target  uncertainties  correspond  to  the  uncertainties  which  are 
generated  in  the  process  of  reducing  a  need  or  military  requirement  into 
cost,  schedule  and  performance  goals.  They  relate  back  to  the  perception 
of  enemy  threat  and  the  resulting  needs  because  as  the  perception  of  the 
threat  changes,  the  needs  and  goals  will  also  change.  Some  specific  areas 
of  uncertainty  which  occur  as  a  result  of  target  uncertainty  include 
uncertainty  in  the  specification  of  the  need,  uncertainty  in  the 
specification  of  desired  operational  capabilities,  uncertainty  in  the 
process  of  generating  the  requirements  of  the  system  and  uncertainty  in  the 
physical  characteristics  that  the  system  must  possess  to  satisfy  the  needs. 
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Internal  program  uncertainties  involve  those  which  originate  within 
the  program  as  a  result  of  the  manner  in  which  the  program  is  organized, 
planned  and  managed.  They  are  not  the  inherent  uncertainties  of  the 
problem  itself  but  are  those  uncertainties  which  are  under  the  direct 
jurisdiction  of  the  program  office.  Examples  of  some  internal  program 
uncertainties  include  the  uncertainties  involved  in  setting  technical,  cost 
and  schedule  targets,  the  uncertainty  involved  in  selecting  the  appropriate 
acquisition  strategy  (e.g.,  how  much  prototyping  is  necessary,  how  much 
testing  is  required,  etc.)  and  uncertainty  in  program  management  which  can 
occur  as  a  result  of  an  improper  balance  between  cost,  schedule  and 

performance . 

Process  uncertainties  are  those  which  originate  outside  the  program, 
but  directly  affect  the  program’s  outcome.  These  uncertainties  are  under 
the  jurisdiction  of  officials  and  agencies  outside  the  program  office  such 
as  the  President,  HQ  USAF ,  industrial  suppliers,  etc.  Some  examples  of 
process  uncertainties  include  uncertainty  in  the  availability  of  funds, 
uncertainty  in  the  availability  of  resources  and  uncertainty  in  the 
priority  assigned  to  the  program.  The  deterministic  values  assigned  to 
these  uncertain  entities  often  form  the  constraints  under  which  the  program 
office  must  work.  The  PO  must  be  able  to  choose  a  flexible  program  which 
is  relatively  constant  under  changes  in  these  uncertain  elements  and  must 

be  ready  to  make  adjustments  as  necessary. 

The  last  major  type  of  uncertainty  in  this  classification  deals  with 
the  question  of  whether  it  is  possible  to  develop  the  desired  system  at  all 
(e.g.,  technical  uncertainty).  It  is  the  smallest  if  the  system  is  within 
the  state-of-the-art.  If  the  system  deals  with  new  technologies  that  have 
yet  to  be  developed,  technical  uncertainty  can  be  very  large  in  the  early 
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phases  o£  the  acquisiciop  process  and  will  hopefully  decrease  as  the 
progras,  advances.  The  assessment  of  technical  uncertainty  can  have  a 
critical  impact  on  the  program  outcome.  If  the  assessment  is  too 
optimistic,  then  the  program  will  be  underbudgeted  and  the  targets  set  by 
the  PO  will  undoubtably  be  exceeded.  These  problems  could  ultimately  lead 
to  program  failure.  An  unduly  pessimistic  outlook  can  be  equally  damaging. 
Such  a  program  may  never  get  budgeted  in  the  first  place. 

From  an  examination  of  Table  1.1,  the  basis  of  several  other 

classification  schemes  can  be  proposed: 

(i)  the  causes  of  uncertainty 

(il)  whether  the  uncertainty  is  caused  by  the  decisions  of  people 
outside  the  PO  or  not 

(iii)  whether  the  uncertainty  comes  from  a  lack  of  information  or  some 
other  uncontrollable  factors. 

(iv)  which  phase  of  the  acquisition  process  the  uncertainty  is  found. 

All  of  the  various  schemes  for  classifying  uncertainty  can  be  useful 
in  breaking  the  problem  into  its  component  parts  and  in  helping  to 
distinguish  between  the  causes  and  the  effects.  The  causes  of  uncertainty 
should  be  carefully  identified,  because  our  developed  risk  assessment  and 
management  framework  is  directed  at  quantifying  the  impact  of  these 
uncertainties  on  the  system  cost,  schedule  and  performance  throughout  the 
weapon  acquisition  process.  The  likelihood  and  impact  of  cost  overruns, 
schedule  slippages,  undesired  system  quality  or  performance,  budget  cuts, 
resource  scarcity  and  program  priority  changes  (i.e.,  the  risks  ) 
have  a  direct  influence  on  the  program  outcome  (see  Figure  1.2).  Their 
identification  and  management  throughout  the  acquisition  process  is  what 
the  following  framework  is  all  about. 

1.2.  Specific  Risk  Assessment  and  Management  Needs  in  WSAj.  A  Framework 
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Since  our  ultimate  aim  is  to  develop  a  comprehensive  risk  assessment 
and  management  aid  for  WSA,  in  this  subsection  we  examine  further  the 
specific  needs  to  which  such  an  aid  must  respond.  Based  on  these 
requirements  and  specifications,  a  master  plan  or  framework  for  developing 
a  WSA  assessment  and  management  system  will  be  subsequently  outlined  in 
Section  3. 

The  type  and  complexity  of  risk  assessment  tools  required  during  the 
weapons  system  acquisition  process  are  dependent  upon  the  phases  of  the 
program.  Uncertainty  is  a  time-related  concept.  The  amount  and  accuracy 
of  information  increases  as  the  program  moves  through  its  phases. 
Normally,  this  additional  data  reduces  many  of  the  uncertainties  in  the 
program.  If  proper  trade-offs  and  decisions  are  made  during  the  course  of 
the  program,  then  the  risk  of  program  failure  should  have  near-zero 
probability  by  the  time  it  reaches  the  production  and  deployment  phase.  If 
as  the  program  progresses  the  risks  cannot  be  reduced,  then  the  program 
either  defies  successful  deployment  or  is  so  prone  to  unforeseen 
difficulties  that  its  successful  completion  is  impossible.  Thus,  it  can  be 
seen  that  the  level  of  program  risk  is  a  function  of  both  the  uncertainty 
in  the  program  and  the  decisions  that  were  made  during  the  course  of  the 
program.  This  means  that  risk  assessment  and  management  procedures  must  be 
tailored  to  the  quality  and  quantity  of  information  present  and  the  types 
of  decisions  which  must  be  made  during  each  of  the  phases. 

The  required  risk  assessment  and  management  procedures  also  depend  on 
the  focus  of  authority.  The  acquisition  process  is  composed  of  several 
hierarchies  of  decision  making  (See  Figure  1.3).  In  this  hierarchical  view 
of  the  acquisition  process,  HQ  USAF  and  higher  authorities  must  make 
about  the  fate  of  each  of  the  different  Air  Force  defense 
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acquisition  programs.  The  program  managers  must  report  to  HQ  USAF  and  make 
operating  and  planning  decisions  related  to  their  individual  programs.  The 
functional  specialists,  which  represent  each  of  the  directorates  or 
divisions  within  the  program  office  (e.g.,  systems  engineering,  program 
contol,  test  and  evaluation,  etc.)  report  to  their  program  manager  and  make 
operating  and  planning  decisions  related  to  their  individual  specialities. 

A  good  risk  assessment  and  management  framework  should  (i)  handle  the 
changing  "risky”  decisions  throughout  the  acquisition  life  cycle  for  the 
different  levels  of  decision  making,  (ii)  help  coordinate  the  flow  of 
available  information  between  the  various  hierarchies  and  subsystems,  and 
(iii)  be  tailored  to  the  "types”  of  risk  decision  needs  and  quantity  and 
quality  of  information  present  in  each  phase  and  hierarchy  of  the 
acquisition  process.  Table  1.2  indicates  some  of  the  typical  types  of 
decisions  which  must  be  made  by  each  of  the  levels  of  authority  during  the 
course  of  a  program.  For  example,  during  Phase  I  (i.e.,  concept 
exploration),  the  main  objectives  for  the  program  office  (the  PM  and 
functional  specialists)  are  to  see  whether  any  of  the  proposed  concepts  are 
feasible  and  to  see  whether  they  satisfy  the  cost,  schedule,  logistic 
supportabili ty ,  and  performance  requirements  levied  by  the  Department  of 
Defense.  In  Phase  II,  the  program  office  must  help  select  the  contractor, 
decide  on  the  criteria  by  which  each  proposal  will  be  evaluated,  and 
whether  additional  iformation  (e.g.,  prototyping)  is  necessary.  During 
Phases  III  and  IV,  the  P.0.  must  direct  the  development,  production,  and 
deployment  of  the  weapon  system,  help  institute  technical  changes,  and 
manage  the  acquisition  of  follow-on  quantities. 

The  types  of  decisions  made  by  HQ  USAF  (and  higher  authorities)  varies 
very  little  during  the  different  phases.  They  normally  consist  of  defining 
and  reevaluating  the  system  needs  and  requirements,  setting  budgets, 


> 


1-9 

schedules,  and  performance  targets,  assigning  priority  levels,  and  deciding 
whether  each  program  should  proceed  to  its  next  phase. 

Risks  must  be  recognized,  quantified,  evaluated,  and  managed  during 
every  phase  and  in  every  level  of  the  acquisition  process  in  order  to  make 
better  -informed  decisions.  Because  we  wish  to  provide  a  useful  tool  for 
risk  assessment  and  management  wi thin  an  acquisition  program,  the  focus  of 
our  risk  assessment  and  management  framework  (and  the  resulting  decision 
support  system)  is  on  the  program  office  (i.e.,  the  program  manager  and  his 
functional  specialists  and/or  directorates).  The  decisions  made  by  HQ  USAF 
and  higher  authorities  are  viewed  as  uncertain  inputs  into  the  PM's 
decision  (See  Figure  1.4.). 

In  order  to  tailor  the  risk  assessment  and  management  framework  to  the 
needs  of  each  of  the  directorates  and  the  program  manager,  we  looked  at  the 
types  of  risk-related  activities  which  are  performed  by  each  of  the 
directorates  (See  Table  1.3  for  a  summary  of  these  activities  throughout 
the  acquisition  process).  The  following  is  a  short  discussion  of  each  of 
the  directorates'  typical  activities: 

program  control --involved  with  planning,  scheduling,  estimating 

budgeting,  analyzing,  and  forecasting  program  progress. 
Identifies  critical  events,  identifies  and  tracks  "program"  risks 
(e.g.,  cost,  schedule,  and  technical  performance),  proposes 
alternative  solutions  to  identified  or  potential  problems,  and 
conducts  "What  if?"  exercises. 

contracting  directorate-sets  award  amounts,  delivery  schedules, 

contract  terms  (i.e.,  contract  evaluation  factors),  and  payment 
arrangements  based  on  technical  and  program  risk  considerations. 
integrated  logistics  support-- involved  in  logistics  planning. 
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Makes  sure  that  support  considerations  (e.g.,  reliability, 
maintenance,  operation,  training,  etc.)  are  included  in  the 
requirements  and  design.  Defines  and  plans  for  support 
requirements  that  are  optimally  related  to  design  and  acquires 
support  during  the  operational  phase  at  minimal  cost. 
Identifies,  tracks  and  proposes  solutions  to  "logistic"  problems 
( risks) . 

manufacturing  management- - involved  in  production  planning.  Makes 

sure  production  feasibility  is  analyzed,  areas  of  "production" 
risks  (e.g.,  in  the  fabrication,  assembly,  installation, 
checkout,  manufacturing  method,  etc.)  are  identified,  and 
proposes  plans  for  potential  and/or  observed  production  problems. 

engineering  management- - involved  in  defining  system  performance  para¬ 
meters  and  configuration,  planning  and  control  of  technical 
tasks,  integrating  reliability,  maintainability,  and  safety,  and 
optimizing  technical  performance  with  cost,  schedule,  and 

logistic  supportabi 1 i ty  to  meet  program  objectives.  Heavily 
involved  in  identifying,  tracking,  and  planning  for  technical 
ri sks . 

test  and  evaluation- - involved  in  the  evaluation  of  the  program's  tech¬ 
nical  and  operational  feasibility.  In  particular,  its  objectives 
are  to  assess  system  specifications,  assess  program  risks/trade¬ 
offs,  assess  logistic  supportability  and  survivability,  verify 
technical  order  completeness,  gather  training  program  and 

environmental  impact  data,  and  determine  system  performance 
limitations . 

configuration  management- -oversees  system  functional  requirements 


from 
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the  subsystems.  In  particular,  they  define  and  verify  the 
configuration  of  items,  control  changes  in  these  items,  monitor 
their  implementation,  and  track  the  configuration  of  units. 
computer  resources  management --oversees  the  development  and  opera¬ 
tion  of  the  computer  subsystem  of  a  weapon  system.  Designs, 
codes,  checks,  tests,  installs,  operates,  and  supports  the 
computer  function.  Identifies  "computer  risks"  and  proposes 
plans  for  potential  and  observed  problems. 

The  risk  activities  identified  in  Table  1.3  provide  some  general 

guidelines  for  choosing  the  appropriate  tools  to  be  included  in  risk 

assessment  and  management  decisions  aids  for  each  of  the  directorates. 

Understanding  the  flow  of  risk  information  between  the  program  office  and 
its  directorates  (which  can  be  summarized  in  diagrams  such  as  Figure  1.5) 
in  each  phase  of  the  acquisition  process  helps  to  identify  information 
requirements  and  provides  the  guidelines  for  coordinating  risk  information 
flow.  These  tables  and  figures  (along  with  an  understanding  of  the 

quantity  and  quality  of  the  risk  information)  form  the  framework  upon  which 
a  risk  assessment  and  management  decision  aid  can  be  designed. 
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Table  1.1.  Categories  of  Uncertainty  in  the  Acquisition  Process  summarized  from 
discussion  of  uncertainty  in  Lenox,  Hamilton  T.  1973. 


1-12 


/.  (  f  <  I  I  1  I  I  '  f  * 


V 


t*t«co  N 

t’lia  -i’  1 

1*1  If  1  so  '? 

Phase  1 

Phase  4 

lint  Ini  <011 

iiliit 

Validol  ion 

(  onccpl 

1  xplondii  it 

iH’inofistriilion 

mid 

Vnlidnl  ion 

1  oil  Scale 

1  >eve|opmenl 

Production 

and 

Deployment 

MQ  USA  1 
and 
higher 
authorit  ies 

"  what  me  the  needs1' 

41  wind  arc  1 1n'  functional  requirements' 

"  wlml  hi  c  Hu*  initial  cost,  schedule 
nnd  performance  largels? 

"  wind  |»r jtfri  1  v  should  In*  given  to  the 

|  »!'0|'.l  J « 1 1» 

"  <%|i<  >i  i  |i  1  iIh*  pi  i  »|*  in  in  In*  funded'4 

"  have*  llio  needs  changed? 

"  <l<<  Hu*  1  urge  In  need  to  ho  adjusted? 

"  should  the  priori  tv  change? 

"  slionlil  1  ho  imdgc  \  hi*  adjusted'7 

41  should  Mu'  program  proceed  lo  llio 
tic  x t  phase  or  In*  nil  '* 

"  Ini vr  1 1 1 <  needs  fliiingcd* 

"  do  1  lie  targets  need  lohe  adjusted’7 

°  should  Ihc  priorilv  change? 

"  should  th('  budget  hr  adjusted? 

"  should  llio  program  proceed  lo  Ihe 
next  phase  or  be  oil  '* 

°  have  llir  needs  changed? 

**  do  the  targets  need  to  lx*  adjusted'7 

Program 

Manager 

(I'M) 

11  is  il  possihh'  to  develop  program 
within  lit)  USAT  targets? 

”  wind  is  llu*  expected  cos|,  schedule, 
pcrfoiiianrc  unit  risks  for  : 

(i  )  development 
(li)  prodiiol  ion 
(tii)  operation 
(iv)  maintenance 
for  ('in'll  concept  '* 

"  who  li  contactor  should  he  selected? 

°  is  prototyping  liecessnrv? 

"  what  is  ||(o  expected  cos|,  schedule, 
porfoi  manor  and  risks  fur; 

(i)  development 
(it)  prodia'iion 
(m)  opernlion 
(iv)  inaiiiloniiner 

for  en<  li  proposal?  i 

"  wind  is  Ihe  planned  budget  and 
schedule? 

°  j<;  i he  contractor  on  schedule  and 
w  i  I  tun  budget  ? 

"  do(*s  the  program  satisfy  technical 

requirements? 

"  are  any  planning  adjust  merits  needed 
for  development,  production  and  de 
plovmeiit,  operation  and  nmmlcii 
ame? 

°  are  any  planning  adjust inenls 
needed  in  htalgel  schedule  or 
performance  for; 

(i)  production  and  deployment 
(il)  operation 
(lii)  nininl  enance 
"  how  should  engineering  changes 
he  incorporated  into  Ihe  system? 

°  how  should  follow  on  items  he 
acquired'7 

1  unci  ionnl 

Specialists 
(directoral es 
or  divisions  of 

1*0) 

i 

"  wind  is  llio  expected  oost  schedule 
mid  permam  c  (mid  risks)  for: 

(i)  d<  volopiiioid 
(li)  prodmlioii 
(dll  operation 
(iv)  maintenance 
(ns  iipproprhdo  for  special!  \  ) 
for  each  com  opt  * 

44  is  more  detailed  information  required 
for  spec  in  It  v? 

1 
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Table  1.2. 


Typical  decisions 
in  each  phase  of 


made  by  the  three  levels  of  authority 
acquisition  program. 


Phase  1 

Concept 

Exploration 

Phase  II 

Demonstration 

and 

Validation 

Pliuse  111 

Full  Scale 

Development 

Phase  IV 

Production 

and 

Deployment 

Remarks 
"risky  areas" 

Program 

Control 

N/A 

N/A 

u  II)  and  track  cost,  schedule  und  per¬ 
formance  risks 

0  propose  alternative  solutions  to  iden¬ 
tified  or  potential  problems 
°  conduct  "whul  if"  exercises 

°  ID  and  track  cost,  schedule  and  per¬ 
formance  risks 

°  propose  alternative  solutions  to  iden¬ 
tified  or  potential  problems 
“  conduct  "what  if"  exercises 

"program  risks" 
cost 

schedule 

performance 

Contracting 

Set  budget,  delivery  schedule,  contract 
terms  and  payment  arrangements  tai 
lored  to  program  risk 

Set  budget,  delivery  schedule,  contract 
terms  and  payment  arrangements  tai¬ 
lored  lo  program  risk 

Set  budget,  delivery  schedule,  contract 
terms  and  payment  arrangements  tai 
lored  to  program  risk 

Set  budget,  delivery  schedule,  contract 
terms  and  payment  arrangements  tai¬ 
lored  to  program  risk 

uses 

"program  risks" 
considerations 

Integrated 

Logistics 

Support 

ID  and  evaluate  support. ibility  risks 
and  impacts  on  program  lor  each  pro¬ 
posed  system  concept 

ID  and  evaluute  snpportability  risks 
and  impacts  on  program  for  each  pro¬ 
posed  system  concept 

°  II)  and  track  supportability  risks 
u  propose  alternative  solutions  to 
identified  or  potential  supportability 
problems 

°  conduct  "wliut  if"  exercises  for  sup- 
portability  considerations 

0  ID  and  track  supportability  risks 

0  propose  alternative  solutions  to 
identified  or  potential  supportability 
problems 

u  conduct  "what  if"  exercises  for  sup¬ 
portability  considerations 

"Logistics  Risk" 
operation 
reliability 
maintenance 
manubiligy 
training  &  skills 
packaging,  handling,  transport 

Manufacturing 

Management 

N/A 

ID  and  evaluute  potential  production 
risks  and  iinpacls  for  cucli  proposul 

determine  production  risks  from  duta 
generated  during  this  phase 

°  ussess  and  Hack  prndulioii  risks 

0  propose  pluns  for  potential  and  obser 
ved  production  problems 

"prodne  ibility"  risk 

fabrication  *“* 

assembly  ^ 

installation  .f> 

checkout 
scheduling 

program  surveillance 

Lngineering 

Management 

0  Define  performance  purameters  and 
confignutions  with  reasonable  techni¬ 
cal  risk 

0  Analyze  technical  risk  of  proposed 
concepts 

w  Define  performance  purameters  and 
confignations  with  reasonable  techni 
cal  risk 

°  Analyze  technical  risk  of  proposed 
concepts 

0  ID  und  truck  technical  risks 

°  optimize  technical  perlot  inuuce 
with  cost,  schedule,  logistic  support- 
ability  (under  risk)  to  meet  objectives 

0  ID  and  track  technical  risks 
°  optimize  technical  performance 
with  cost,  schedule,  logistic  support 
ability  (under  risk)  to  meet  objectives 

"tcclinicul  risk" 
technical  design 
reliability 
maintenance 
safety 

Test  and 

Evaluation 

Evaluate  technical  and  operational 
risks  for  each  proposed  concept  (work 
ing  with  other  diretorates] 

Evaluate  technical  and  operational 
risks  for  cucli  proposed  concept  (work 
ing  with  other  diretorates) 

u  ID  and  track  risks 

0  correct  and  plan  ugainsl  technical 
deficiencies 

0  asses  progruni  risks/ tradeoffs 

°  evaluute  system  under  uncertain 
(ehunging)  environment 
°  evaluate  risks  in  system  changes 
°  assess  program  risks/tradeoffs 

"Performance  Risks" 
environmental  impact 
technical  performance 
operational 
needs  effectiveness 
supportability 
survivability 

Configuration 

Management 

define  "design  to"  specification  using 
risk  consideration  for  entire  system 

define  "build  to"  specifications  using 
risk  considerations  for  each  configur¬ 
ation  item  (subsystem) 

define  "build  to"  specifications  using 
risk  considerations  for  each  configur¬ 
ation  item  (subsystem) 

N/A 

uses 

"technical  risk" 
considerations 

Computer  Resources 
Management 

ID  und  evaluate  potcntiul  computer 
risks  for  eaeh  concept 

ID  and  evaluate  potential  computer 
risks  for  each  concept 

°  ID  ami  track  computer  risks 

0  propose  alternutive  solutions  for  po¬ 
tential  problems 

0  ID  and  track  computer  risks 

0  propose  alternative  solutions  for  po¬ 
tential  problems 

"computer  risks" 
design 
coding 
instalfation 
operation 

Program 

Manager 

Summarize  all  risks  and  impacts  of 
eacli  proposed  system  concept 

Summarize  technical,  cost,  support  ab¬ 
ility,  production  and  schedule  risks  and 
impacts  for  each  proposed  system 

0  Evaluute  and  manage  all  development 
risks 

0  Choose  appropriate  uction  for  iden¬ 
tified  problems  using  risk  considera¬ 
tions 

0 Evaluate  and  manage  production  and 

deployment  risks 

°l  valuatc  und  manage  technical  and 
cost  risks  when  engineering  changes 
are  required 

°  Evaluate  and  inunage  program  risk 
when  acquiring  follow-on  quatities 

all  previous  risks 

* 

Table  1.3.  Typical  Risk  Related  Activities  for  the  Program  Manager  and  his  Directorates 
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IMmsc  0 

Initiation  and 
Validation  of 
Program 


Needs  identified 
SON 

Review  ami 
approval  process 

_  Ihidget  ii 

lg  review 

I - 

no 

f  unds 
j  Ponds 

STOP  approved 


Phase  1 

Concept 

Exploration 


Program  office 
established 

< 

t 

Request  for  proposals 
sent  out 

- 1 

proposals 

Proposals  reviewed 
by  PO  j 

i _ _ 

1)01)  ex 
progruu 

terfml 
i  review 

nmuf 

Phase  2 

1  )einonstrat  ion 
and 

Validation 


Request  for  more 
detailed  proposal* 
sent  out 

r~ 


PO  source  selection 
process 


no  feas 
it)le  op 
tionr. 

I 

STOP 


cancelled 

I 

STOI* 


program 

approved 


>rogram  review 


progra  m 
approved 


Phase  3 

Full  Seale 
Development 
(I  Sl» 


program 

cancelled 

1 


Phase  4 

Production 

and 

I  )eployment 


System  development  ♦ 
and  testing 

3 

t _ 

♦  PO  internal  program 
reviews 

3 

£ - 

*  1)01)  external 
program  review 

< 

program 

approved 


more 

informa 

tion 


Items  produced  n 


and  deployed 


PO  internal 
program 
review 


“1 

prograi 
icelle 

I 

STOP 


T. 


program 

changes 

_ I 


target 

changes 


1)01)  external 
program  review 


Program 

Completed 


program 

cancelled 

I 

STOP 


*  indicates  where  risk  assessment  and  management  is  required  in  PO 
*♦  indicates  where  risk  assessment  and  iimmigeincnt  is  required  in  1)01) 


Figure  1.1.  Simplified  view  of  acquisition  process 
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Figure  1.2. 


Relationship  between  uncertainties,  risks  and  program  failure  in 
weapon  system  acquisition  process. 
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for  example: 

1)  systems  engineering 

2)  configuration  management 

3)  program  control 

4)  management  support 

5)  contracting 

6)  manufacturing  management 

7)  test  and  evaluation 

8)  integrated  logistics  support 


Figure  1.3. 


Levels  of  authority  in  the  weapon  system  acquisition  process. 
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Figure  1.4. 


Focus  of  risk  assessment  and  management  framework 
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Figure  1.5.  Typical  information  flow  in  risk  assessment  and  management 
framework  during  Phase  III. 
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2. 


Ri sk  Assessment  and  Management  Methodology 


The  previous  section  focuses  on  the  specific  needs  and  requirements  of 
the  WSA  risk  assessment  and  management  problem.  In  this  section,  we 
discuss  what  tools  are  available  or  need  to  be  developed  to  respond  to  such 
needs.  In  particular,  we  describe  a  general  risk  assessment  and  management 
philosophy,  summarize  some  selected  risk  analysis  tools,  and  propose  a 
methodological  framework  for  integrating  various  tools  and  concepts  that 
reflect  our  vision  of  the  ultimate  decision  support  system.  We  shall  be 
brief  in  this  section.  Appendix  B  provides  a  more  detailed  discussion  and 
review  of  risk  assessment  and  management  methods. 

2.1.  Risk  Assessment  and  Management  Process 

Risk  is  the  possibility  of  suffering  harm,  loss,  danger,  failure,  or 
some  kind  of  adverse  effects  as  a  result  of  taking  an  action  or  a  sequence 
of  actions.  There  are  thus  two  basic  elements  associated  with  risk:  the 
magnitude  and  the  1 ikelihood  of  harm  or  adverse  effects.  To  describe  a 
risky  situation,  we  must  therefore  adequately  describe  these  two  basic 
elements.  Risk  is  clearly  induced  by  some  uncertainties  and  these  may  be 
of  different  types.  Some  uncertainties  are  caused  by  natural  random  or 
uncontrollable  forces  from  outside  the  system  or  context,  and  these  are 
naturally  uncontrollable  by  any  means  generated  within  the  system. 
Precipitation,  wind,  and  earthquakes  are  but  a  few  examples  of  such  forces. 
In  the  WSA  process  as  viewed  from  the  program  office's  perspectives, 
decisions  and/or  directives  from  the  command  level  or  higher  may  also  be 
viewed  as  uncontrollable  factors  (uncontrollable  from  within  the  program 
office)  and  can  be  a  major  source  of  uncertainty  to  the  program  manager's 
domain  of  operation.  Another  type  of  uncertainty  arises  from  lack  of 
information  and  may  have  nothing  to  do  with  any  natural  random  factors.  A 
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tank  commander  in  a  battlefield  may  wish  to  know  the  enemy  strength  in 
order  to  develop  an  appropriate  attack  strategy.  Although  there  is  nothing 
naturally  random  about  the  enemy  strength,  any  estimate  he  uses--which  will 
only  be  as  good  as  the  quantity  and  quality  of  intelligence  information  he 
has  on  hand--will  usually  be  a  source  of  uncertainty,  and  hence  of  risk. 
Between  the  two  extremes  are  uncertainties  that  are  caused  by  both 
mechanisms . 

To  perform  the  complete  process  of  risk  assessment  for  a  particular 
problem,  the  following  tasks  need  to  be  carried  out  (see,  for  example,  Rowe 
[1981]): 

1 )  Risk  identification ,  which  involves  identification  of  the 
nature,  types,  and  sources  of  risks  and  uncertainties.  In  general, 
the  major  types  of  risks  are  financial,  health-related,  environmental, 
and  technical  (e.g.,  performance  and  suppor tab ili ty ) .  The  end 
products  of  this  task  are  a  complete  description  of  risky  events  and 
elements  of  major  concern  along  with  their  causative  factors  and 
mechanism . 

2) Risk  quantification ,  which  entails  formulating  appropriate  measures 
of  risk  and  estimating  the  likelihood  (probability)  of  occurrence  of 
all  consequences  associated  with  risky  events  as  well  as  the  magnitude 
of  such  consequences. 

3) Risk  evaluation ,  which  includes  selection  of  an  evaluation  procedure 
(e.g.,  optimizing  expected  value,  trade-off  analysis)  and  analysis  of 
various  possible  impacts  of  risky  events. 

4) Risk  acceptance  and  aversion,  which  requires  decision  making 


regarding  both 

an 

acceptable 

level 

of  risk, 

and  its  equitable 

distribution . 

This 

phase  of 

risk 

assessment 

also  involves  the 

development  of 

risk 

control  (i, 

.e.  , 

measures  to 

reduce  or  prevent 
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risk) . 

3 ) Risk  management,  which  involves  the  formulation  of  policies,  the 
development  of  risk  contol  options  (i.e.,  methods  to  reduce  or  prevent 
risk),  and  execution  of  such  policy  options. 

2 . 2 .  What  are  Availab le 

There  are  a  number  of  tools  which  have  been  developed  to  perform  one 
or  more  tasks  in  the  overall  risk  assessment  and  management  process.  These 
tools  are  indeed  diverse  in  nature,  emphasis,  purpose,  and  degree  of 
comprehensiveness  and  sophistication.  Appendix  B  reviews  a  large 
collection  of  these  tools  based  on  several  criteria.  It  is  evident  from 
the  review  that  no  single  tool  can  adequately  support  all  tasks  that  need 
to  be  done  in  the  overall  process  of  risk  assessment  and  management.  Nor 
can  any  one  method  claim  to  be  a  general  purpose  procedure  that  can  deal 
with  all  types  of  risk  (financial,  health,  environmental  and  technical)  and 
in  all  types  of  risky  situations.  For  example,  work  breakdown  structure 
(WBS)  should  be  a  very  useful  device  for  identifying  risky  elements 
associated  with  costs.  By  itself  however,  it  is  not  set  up  for  risk 
evaluation  purposes.  Cost-benefit  analysis,  on  the  other  hand,  is  an 
evaluation  tool  used  to  appraise  various  alternatives  based  on  monetary 
measures.  It  is  not  suitable  for  risk  identification  purposes.  As  another 
example,  consider  the  well-known  multiattribute  utility  theory  approach. 
Such  a  method  may  be  an  appropriate  risk  evaluation  tool  for  a  class  of 
problems  with  moderate-to-high  frequency  of  risk  and  moderate- to- low 
damage.  However,  they  may  fail  to  capture  the  possible  devastating  effects 
of  the  low-frequency,  high-damage  characteristics  typical  in  extreme 
events.  Here,  as  a  typical  expected-value  approach,  extreme  events  with  a 
low  probability  of  occurrence  are  given  the  same  proportional  weight  and 
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importance  (in  the  multiobjective  commensurate  process)  regardless  of  their 

potential  catastrophic  and  irreversible  impact.  Yet  it  is  a  commonly 

acknowledge  fact  that  the  outcome  of  a  catastrophic  accident  that  may  cause 

-5 

10,000  deaths  with  a  low  frequency  of  10  is  neither  perceived  nor 

accepted  to  be  in  the  same  category  of  more  common  accidents  that  occur 

-1 

with  a  much  higher  frequency  of,  say,  10  ,  but  may  cause  the  death  of  one 

person  each  time. 

Because  of  the  diversity  of  risk  problems  that  may  arise  and  of  the 
tasks  that  need  to  be  done,  a  general  purpose  method,  even  if  it  can  be 
developed,  would  be  almost  without  content  and,  thus,  will  most  likely  be 
useless.  What  is  needed  is  therefore  an  ensemble  of  tools  that 
collectively  span  the  whole  spectrum  of  risk  assessment  and  management 
tasks  for  a  particular  problem  encountered. 

In  the  WSA  problem,  typical  types  of  risk  emphasized  are  financial 
risks  (cost  overrun,  budget  cuts,  schedule  slippage,  etc)  and  technical 
risks  (e.g.,  substandard  performance,  supportability ,  etc.).  A  possible 
collection  of  tools  that  may  be  useful  for  developing  a  comprehensive  WSA 
risk  assessment  and  management  system  will  be  identified  subsequently. 

2.3.  What  is  Needed :  An  Integrated  Methodological  Framework 

Weapon  system  development  and  acquisition,  as  a  project,  normally 
evolves  through  various  phases  in  the  time  dimension  to  complete  its  life 
cycle.  These  phases  include  project  initiation,  conceptual  exploration, 
demonstration  and  validation,  full-scale  development,  production  and 
deployment,  and  retirement.  Each  phase  consists  of  many  interrelated 
activities  and  tasks  to  be  performed,  and  a  large  number  of  interconnected 
decisions  to  be  made  in  an  environment  filled  with  different  levels  of 
uncertainties.  The  program  manager  has  the  responsibility  of  making  many 
of  these  critical  decisions  and  of  managing  the  project  to  ensure  efficient 
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and  effective  progress  toward  completion. 

To  cope  with  the  inherent  complexity  and  multitide  of  uncertainties, 
there  is  clearly  a  need  for  an  integrated  and  comprehensive  decision  aid 
that  will  allow  the  program  manager  to  make  more  informed  decisions  in  a 
timely  and  efficient  manner.  Such  a  decision  aid  should  i)  provide  the 
program  manager  a  well-balanced  picture  of  the  project,  ii)  allow  the 
pro  gram  manager  to  access  various  data  bases  quickly  and  conveniently  and 
iii)  furnish  an  ample  selection  of  decision/risk  analysis  tools  that 
collectively  cover  a  wide  variety  of  anticipated  decision  and  risk 
assessment  situations.  In  response  to  such  a  need,  we  propose  the 
development  and  use  of  a  decision  support  system  (DSS).  The  general 
philosophy  and  concept  of  DSS  as  well  as  its  specific  content  suitable  for 
decision  making,  and  risk  assessment  and  management  in  weapon  system 
acquisition  are  dicussed  in  the  next  section. 

Since  we  perceive  weapon  system  acquisition  as  a  project  and  the 
concerns  of  the  program  manager  are  planning,  management  and  control  of  the 
project,  network-based  methodologies  appear  to  be  natural  and  logical 
candidates  to  choose  from. 

Cursory  inspection  of  a  typical  weapon  system  acquisition  project 
reveals  that,  despite  its  appearance,  the  program  structure  is  far  from 
being  rigid.  The  program  is  marked  with  considerable  uncertainties. 
First,  it  contains  a  number  of  decision  points  whose  outcomes  cannot  be 
taken  for  granted.  Activities  succeeding  these  decision  points  which 
operate  under  the  assumption  of  getting  favorable  decision  outcomes  in  the 
present  program  plan,  may  need  a  back-up  plan  should  a  decision  outcome  be 
otherwise.  Some  activities  also  have  a  probability  of  not  being  performed 
in  the  present  format  due  to  the  likelihood  of  unforeseen  circumstances. 
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This  may  necessitate  consideration  of  alternative  activities  and 
strategies.  Network-based  techniques  that  can  handle  less  rigid  structure 
thus  appear  to  be  quite  appropriate.  Of  these,  GERT  and  VERT  stand  out  as 
they  facilitate  formal  and  comprehensive  quantitative  risk  assessment  in 
terms  of  project  time,  cost  and  performance.  SCERT,  on  the  other  hand, 
furnishes  opportunities  to  deal  with  risk  aspects  associated  with  other 
factors,  but  mostly  in  a  qualitative  and  less  formal  fashion.  Moreover, 
SCERT  is  still  in  a  developmental  stage,  although  preliminary  versions  have 
been  successfully  applied  to  some  large  engineering  problems. 

The  network-based  techniques  just  mentioned  mostly  rely  on  simulation 
to  generate  the  magnitudes  and  associated  statistics  about  project 
completion  data,  project  cost  and  performance.  Except  for  SCERT,  no 
specific  mechanism  is  provided  to  combine  this  information  in  an 
appropriate  way  to  aid  the  decision  maker  in  making  the  final  decisions. 
This  is  where  we  envision  such  techniques  as  decision  analysis  and 
multiobjective  risk  trade-off  (discussed  in  Appendix  B)  playing  a 
significant  role.  Outputs  from  network-based  procedures  such  as  GERT  or 
VERT  can  be  used  as  a  basis  for  formulating  appropriate  risk  measures  which 
can,  in  turn,  be  used  for  multiobjective  risk  trade-off  analysis  in  the 
same  spirit  as  discussed  in  Appendix  B. 

With  this  preliminary  impression  of  the  weapon  system  acquisition,  we 
envision  an  interactive  decision  support  package  combining  an  appropriate 
tool  in  the  network  class  with  one  of  the  multiobjective  risk  trade-off 
analysis  tools  (or  decision  analysis)  as  a  potentially  promising  system  to 
satisfy  the  program  office  needs.  Such  a  package  should  be  developed  so 
that  it  i)  is  user-friendly,  ii)  has  great  flexibility  and  the  capability 
of  handling  a  wide  range  of  decision  situations  with  respect  to  the  weapon 
system  acquisition,  iii)  has  a  quick  turn-around  time  so  that  an  emergency 
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decision  can  be  made,  iv)  is  built  on  a  modularity  concept,  where 
components  of  the  system  can  be  modified  or  replaced  without  the  need  to 
change  the  entire  system,  v)  has  the  capability  of  handling  the 
hierarchical  decision-making  structure  of  the  program  office,  and  vi)  has 
the  capability  of  quantifying  the  impacts  of  ecisions  made  at  time  t  =  k 
on  the  system  at  a  later  time,  say  t  =  k  4*  n.  The  MMIAM,  for  example,  is 
a  method  well-suited  to  this  purpose. 

Although  the  packaging  will  be  done  in  the  form  of  a  DSS,  Figure  2.1 
shows  interrelationships  of  the  DSS  components  along  the  large  dimension 
(e.g.,  logical  steps). 
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Figure  2.1  A  Procedural  Framework  for  WSA  Risk  Assessment  and  Management 
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3.  A  Decision  Support  System  for  Weapon  System  Acquisition 
Risk  Management 

In  the  previous  section,  we  identified  a  decision  support  system  (DSS) 
as  a  logical  choice  for  packaging  various  risk  analysis  modules  into  one 
integrated  system.  Since  it  will  play  such  a  central  role,  we  will  now 
describe  the  underlying  philosophy  and  basic  structure  of  a  typical  DSS. 
Specific  components  of  the  WSA  risk-assesment  DSS  that  we  envision  will 
then  follow. 

3.1  What  is  a  DSS? 

A  decision  support  system  (DSS)  is  an  intelligent  interactive  man- 
machine  decision  aid.  It  helps  decision  makers  make  more  informed 
decisions  faster  by  allowing  them  i)  to  get  quickly  access  to  multiple 
data  bases,  and  ii)  to  perform  sophisticated  data  processing  and  system 
analysis  techniques  with  great  speed,  accuracy,  and  efficiency.  The  DSS 
can  also  be  tailored  to  the  particular  skills  and  needs  of  the  user.  This 
quality  of  a  DSS  is  particulary  useful  for  the  computer  novice  or  people 
who  have  little  or  no  background  concerning  the  technical  nature  of  the 
models  or  analyses  utilized  (e.g.,  network  modeling  and  risk  analysis)  but 
would  greatly  benefit  from  the  information  such  models  and  analyses  could 
provide . 

The  DSS  proposed  here  is  envisioned  as  a  tool  to  help  improve 
strategic,  managerial,  and  operational  decision  making  throughout  the 
weapon  system  acquisition  process.  This  DSS  will  be  tailored  to  all  phases 
of  the  acquisiton  process  and  will  possess  the  following  attributes: 

(i)  ease  of  use--does  not  require  expertise  in  the  area  of  program 
scheduling  or  risk  assessment 

(ii)  produces  output  usable  by  the  program  manager 

(iii)  is  tailored  specifically  for  the  Department  of  Defense  (DOD) 
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weapon  system  acquisition  process 

(iv)  integrates  network  creation,  schedule  management,  risk 
assessment,  and  trade-off  and  impact  analyses 

(v)  is  designed  for  use  on  microcomputers--! . e . .  the  IBM-XT 
A  prototype  DSS  with  the  above  attributes  has  been  developed  in  this 
project.  Its  limited  scope  for  the  production  and  deployment  phase  can  be 
extended,  however,  to  a  DSS  with  a  broader  range  of  capabilities  and  wider 
applicability  to  encompass  all  phases  of  the  weapon  system  acquisition 
process . 

3 . 2  Why  a  Deci s ion  Support  System? 

Making  decisions  about  large  and  complex  problems  is  a  difficult  task. 
Until  recently,  decisions  concerning  large-scale  problems  have  mainly  been 
based  on  intuition  coupled  with  experience,  ingenuity,  and  value  judgment 
(possibly  aided  by  verbal,  but  logical,  reasoning),  simple  calculations, 
and  simple  graphic  devices,  such  as  graphs  and  charts.  However,  in  today's 
society,  as  more  and  more  demands  are  placed  upon  our  normally  limited 
resources  (from  the  combined  effects  of  new  societal  needs  and  the  rapid 
advance  of  technology),  it  becomes  obvious  that  more  efficient  and 
effective  decision-making  approaches  are  needed.  Two  basic  premises 
underline  modern-day  decision  making: 

(1)  The  decision  maker's  wisdom  and  value  judgment  can  never  be 
replaced  by  any  completely  mechanistic  process. 

(2)  The  decision  maker  will  generally  make  a  better  decision  if 
he/she  is  well  informed  about  relevant  aspects  of  the  systems 
within  which  he/she  is  making  decisions. 

The  first  premise  implies  that  whatever  decision  tools  are  employed 
and  whatever  their  sophistication,  the  decision  maker(s)  must  still  form  an 
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integral  part  of  the  decision-making  process--at  least  in  the  final  stage 
of  the  process.  The  second  premise,  on  the  other  hand,  implies  that  before 
an  important  decision  is  made,  a  system  should  be  systematically  analyzed 
in  sufficient  detail  that  relevant  results  can  be  presented  to  the  decision 
maker.  As  the  human  mind  has  a  limited  capacity  to  process  and  comprehend 
large  amounts  of  information  at  any  one  time,  careful  selection  of  results 
is  needed  to  avoid  overwhelming  the  decision  maker  with  too  much 
information.  Also,  great  care  should  be  taken  in  the  methods  of  presenting 
the  information.  Furthermore,  the  credibility  and  accuracy  of  all 
information  should  be  constantly  checked  to  avoid  possible  ill-fated 
decisions . 

To  satisfy  the  above  requirements,  a  good  decision-making  procedure 
for  large  and  complex  problems  should  possess  the  following  desirable 
characteristics : 

1)  a  good  data  base--to  ensure  credibility  and  accuracy 

2)  a  fast,  efficient,  and  comprehensive  process  of  transforming  data 

into  intelligent  information-- to  ensure  that  various  relevant 
factors  are  properly  taken  into  account 

3)  an  effective  mechanism  for  interacting  with  the  decision  maker 

through  a  decision  support  system. 

A  DSS  furnishes  a  means  of  achieving  such  goals  while  providing  great 
flexibility  in  utilizing  data  bases  and  models  in  a  convenient  and  easy-to- 
use  format.  In  the  context  of  the  acquisition  process,  a  well-designed  DSS 
would  aid  the  program  manager  with  all  of  the  decisions  he  must  make 
throughout  a  program's  duration.  In  particular,  during  the  early  phases  of 
the  acquisition  process  (i.e.,  concept  exploration  and  demonstration  and 
validation),  program  or  process  selection  decisions  are  based  on  data  at 
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the  primitive  level  (data- focus ) .  During  the  final  phases  of  the 
acquisition  process,  the  operating,  planning,  and  management  decisions  are 
based  on  more  intelligent  information,  which  is  obtained  by  passing 
primitive  data  through  some  analytical  models  (e.g.,  network  models,  trade¬ 
off  analyses,  risk  analyses,  etc.).  A  properly  designed  DSS  would  allow 
for  this  flexible  usage  of  various  data  bases  and  models  according  to 
specific  needs,  with  fast  turn-around  time,  thus  providing  a  convenient 
mechanism  for  mixing  the  value  elements  and  factual  elements  to  help  the  PM 
make  decisions  in  the  most  effective  way. 

3 . 3  Components  of  a  Decision  Support  System 

Advances  in  DSSs  have  increased  rapidly  with  the  development  of  more 
powerful  and  sophisticated  mini-  and  micro-computers.  These  new  computers 
have  allowed  for  improved  interaction  between  the  system  and  user  and  thus 
providing  greater  power,  flexibility,  and  ease  of  use  of  the  DSS.  From  the 
DSS  user’s  point  of  view,  these  interaction  capabilities  between  the  system 
and  the  user  (i.e.,  the  dialog  component)  encompass  the  entire  system. 
According  to  Bennett  [1977]  the  dialog  ’’experience"  can  be  divided  into 
three  parts: 

(i)  The  action  language--wha t  the  user  "can  do"  in  communicating  with 
the  system.  It  includes  such  options  as  the  keyboard,  touch 
panels,  joy  stick,  voice  command,  mouse  functions,  and  any  other 
inputting  devices. 

(ii)  The  display  (or  presentation)  language- -what  the  user  "sees." 
The  display  language  includes  options  such  as  type  of  character 
or  line  printer,  display  screen,  graphics,  color  capabilities, 
plotter,  audio  capability,  and  any  other  outputting  processes  or 
devices . 


( iii )  The  knowledge  base--what  the  user  "must  know." 


The  knowledge 
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base  consists  of  all  the  things  that  the  user  needs  to  bring  with 
him  to  a  session  with  the  system  in  order  to  use  the  DSS 
effectively. 

Although  the  user  only  works  with  the  dialog  component  (i.e.,  action 
language,  display  language,  and  knowledge  base),  it  is  obvious  that  there 
is  more  to  a  DSS  then  just  this.  A  decision  support  system  is  a 
combination  of  (i)  optimization,  simulation,  and  heuristic  models  (the 
model  base),  (ii)  an  extensive  data  base,  and  (iii)  an  information 
management  system  which  handles  the  dialog  between  user,  models,  and  data. 
Figure  3.1  summarizes  the  relationships  between  these  three  components. 

The  model  base  can  be  considered  to  be  the  brains  of  the  system.  It 
allows  the  user  to  retrieve,  manipulate  and  evaluate  the  information 

available  to  him  through  the  data  base.  The  data  base  not  only  contains 
numerical  information  but  can  also  contain  functional  relationships  between 
data  items  and  other  descriptive  information  about  the  problem  at  hand. 
The  capabilities  of  the  management  system  include  model  base  management 

(MBM) ,  data  base  management  (DBM),  and  dialog  generation  and  management 
(DGM) .  The  DGM  system  (DGMS)  provides  the  interactive  interface  between 
the  user  and  the  system,  which  is  what  makes  the  DSS  such  a  powerful  tool 
in  decision  making. 

3.4  DSS  as  a  Packaging  Tool  for  Ri sk  As sesment  and  Management  in  WSA 

In  Section  1  of  this  report,  we  noted  that  a  good  risk  assessment  and 

management  decision  aid  should  address  the  specific  needs  of  the  program 
manager  and  each  of  his  directorates  and  that  the  decision  aid  should  also 
be  tailored  to  the  quality  and  quantity  of  information  available  in  each  of 
the  acquisition  phases.  In  order  to  aid  the  program  manager  and  the 
directorates  in  making  program  decisions,  the  decision  aid  should  have  the 
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following  capabilities  (adapted  from  Ingalls,  1984b). 

1)  Network  generation  capability--It  should  have  an  easy-to-use 

process  for  network  generation  which  requires  a  minimum  of  user 

inputs.  The  network  model  should  allow  for  easy  tiering  and 
interaction  of  subnetworks--nhierarchical  networking.”  The 

networks  developed  by  the  lower  levels  (i.e.,  the  functional 

specialists  or  directorates)  should  represent  activities 

controlled  at  the  top-level  of  the  network  (i.e.,  by  the  program 
manager) . 

2)  Schedule  management  capability--It  should  be  able  to  identify 

the  critical  path,  allow  the  user  to  update  the  program  as  it 
proceeds,  and  allow  the  user  to  easily  perform  ”What  if?” 
exercizes  by  inputting  changes  and  receiving  feedback  on  schedule 
impact  s . 

3)  Risk  identification  capability--It  should  provide  a  systematic 

procedure  (possibly  in  the  form  of  a  work  breakdown  structure) 
that  will  identify  risks  throughout  the  acquisition  process. 
Each  directorate  should  have  its  own  version  of  this,  and  the 
decision  aid  should  be  able  to  compress  the  information  from 
these  individual  work  breakdowns  into  a  form  useful  to  the 
program  manager. 

4)  Risk  quantification  capability- - It  should  provide  easy-to-use 
procedures  for  quantifying  risks.  A  number  of  different 
procedures  should  be  available  (e.g.,  subjective  probability 
tools,  cross-impact  analysis,  cost  estimating  techniques, 
reliability  analysis,  etc.),  each  tailored  to  the  quality  and 
quantity  of  the  available  information. 

5)  Risk  evaluation  capabi lity--It  should  allow  the  user  to  easily 
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input  uncertainty  parameters  into  the  network  and  estimate  cost 
and  schedule  risks  for  use  by  the  program  manager  and  each  of  his 
directorates.  It  should  also  allow  for  analysis  of  system 
performance  risks  and  program  failure  risks  through  procedures 
such  as  fault  tree  analysis,  event  tree  analysis,  Bayesian 
decision  analysis  and  an  enhanced  form  of  network  analysis. 

Risk  management  capabil ity--It  should  help  the  decision  maker 
select  an  appropriate  "risky11  plan  of  action  through  procedures 
such  as  multiple-objective  optimization  (e.g.,  the  surrogate 
worth  trade-off  method,  Haimes  et  al .  1975),  trade-off  analysis, 
and  impact  analysis  (such  as  the  multiple-objective  multistage 
impact  analysis  method  of  Gomide  and  Haimes,  1984). 

A  large  number  of  tools  and  methodologies  in  the  areas  of  network 
creation,  schedule  management,  risk  assessment,  and  risk  management  are 
curently  available.  These  are  summarized  in  Section  2  of  this  report. 
However,  in  their  current  form,  each  of  these  tools  and  methodologies 
suffers  from  a  number  of  deficiencies: 

1)  difficult  to  use--designed  for  use  by  "experts,"  or  input 
requirements  are  prohibitive 

2)  output  not  suitable  for  use  by  PO--must  be  translated  by  an 
"expert"  before  it  is  in  a  form  that  can  be  understood 

3)  designed  for  a  different  application-- terminology  and  parameters 
are  inappropriate  for  weapon  system  acquisition  risk  management 

4)  do  not  intgrate  the  six  basic  capabilities  previously  identified 

5)  cannot  be  used  on  a  microcomputer--designed  for  use  on  a 
mainframe  computer. 

A  wel 1 -de signed  decision  support  system  would  correct  each  of  these 
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deficiencies.  In  particular,  by  modifying  and  tailoring  some  of  the 
currently  available  programs  (e.g.,  network  models,  SWT,  PMRM,  etc.)  to 
acquisition  risk  management  needs,  improving  their  user  interaction 
capabilities,  and  developing  some  additional  risk  tools  (e.g.,  a  modified 
work  breakdown  structure),  it  would  be  possible  to  design  a  decision 
support  system  that  will  work  on  a  microcomputer  with  hard  disk  storage 
(e.g.,  IBM-XT) . 

3.5.  Examination  of  the  Risk  Assessment  and  Management  DSS 

The  list  of  six  desired  capabilities  for  a  risk  assessment  and 
management  decision  support  system  given  in  Section  3.4  allows  us  to 
identify  a  number  of  useful  classes  of  tools  to  be  included  in  the  DSS: 

1)  extension  of  work  breakdown  structure  and  method  of  moments 

2)  subjective  probability  tools 

3)  cross-impact  analysis 

4)  cost  estimation  techniques 

5  heuristics,  variance  methods,  charts  and  graphs 

6)  reliability  analysis 

7)  network  modeling  and  network  analysis 

8)  analysis  using  the  probability  tree,  decision  tree,  fault  tree 
and  event  tree 

9)  Bayesian  decision  analysis 

10)  multiple-objective  optimization, ,  trade-off  analysis  and  impact 
analysis . 

From  the  framework  developed  in  Section  1  of  this  report,  we 
identified  the  phases  in  which  each  of  these  tools  would  probably  be  of  the 
most  use  and  by  whom.  Tables  3.1  and  3.2  summarize  these  results  and 
provide  the  guielines  for  designing  the  DSS  model  base  for  the  program 
office  and  each  of  its  directorates.  The  model  base  management  system 
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would  select  the  appropriate  tools  and  methodologies  and  tailor  them  to  the 
needs  and  specifications  of  each  of  the  directorates  and  the  program  manager 
during  each  of  the  phases  of  the  acquisition  program. 

The  data  base  for  the  DSS  would  be  composed  of  the  hierarchical 
networks,  work  breakdowns,  and  tree  diagrams  for  each  of  the  directorates. 
The  data  base  management  system  would  coordinate  the  information  sharing 
between  the  directorates'  and  the  program  manager's  data  and  would  provide 
the  means  to  accomplish  data  consolidation  for  the  higher  levels  in  he 
hierarchies  (e.g.,  the  program  manager  or  the  head  of  he  directorate  if  a 
directoate  needs  to  be  divided  into  subsystems  for  easier  handling). 

The  dialog  management  system  would  provide  all  of  the  "nice"  features 
of  a  good  DSS,  including: 

1)  color  graphics  capability 

2)  use  of  the  "mouse"  for  input/output 

3)  hard  copies  for  all  graphs,  charts,  and  diagrams 

4)  progress  report  and  bar  chart  generation  with  user  interaction 

5)  intelligent  screen  layout  for  data  input 

6)  menu-driven  subroutines  for  each  of  the  directorates  and  the 
program  manager 

7)  help  messages,  error  alerts,  etc. 

Each  of  the  described  components  of  the  DSS  for  risk  assessment  and 
management  (See  Figure  3.1)  would  be  available  on  a  microcomputer  with  hard 
disk  capability  (e.g.,  IBM-XT)  and  written  in  a  common  high-order  language 
(e.g.,  Pascal  or  FORTRAN).  A  system  with  all  of  the  capabilities  we  have 
mentioned  should  be  of  great  use  throughout  a  weapon  system  acquisition 
program . 


In  order  to  demonstrate  a  number  of  features  that  we  envision  for  the 


larger  DSS,  we  developed  a  limited  prototpye  (which  could  be  utilized  as  a 


module  in  the  larger  DSS)  for  use  by  the  program  manager  during  pha 
and  IV  of  the  acquisition  process.  A  detailed  description  of  this 
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Cross 
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Heuristics 

variance  methods 

charts 
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Reliability 

analysis 

Network  modeling 

Network  analysis 

Probability, 
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Bayesian 

decision 

analysis 

Single  and  multiple 

objective  optimi¬ 

zation,  trade-off 
analysis,  and  im¬ 

pact  analysis 
(MMIAM) 

Remarks 

Program  Control 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Identifies,  quantifies,  evaluates  and 
manages  program  risks 

Contracting 

X 

X 

Uses  risk  considerations  in  developing 
contract  terms 

Integrated 

Logistics  Support 

X 

X 

X 

X 

X 

X 

X 

X 

:  x 

X 

Identifies,  quantifies,  evaluates  and 
manages  logistics  risks 
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Manufacturing 

Management 

X 

X 

X 

X 

X 

X 

X 

X 
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Identifies,  quantifies,  evaluates  and 
manages  production  and  manufacturing 
risks 

Engineering 

Management 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Identifies,  quantifies,  evaluates  and 
manages  technical  developmental  risks 

Test  and 

Evaluation 

X 

X 

X 

X 

X 

X 

X 

X 

Identifies,  quantifies  and  evaluates  all 
risks  related  to  system  performance 

Configuration 

Management 

X 

X 

X 

Uses  risk  considerations  in  developing 
configuration  designs 

Computer  Resources 
Management 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Identifies,  quantifies,  evaluates  and 
manages  all  computer  risks 

Table  3. 1  Usage  Breakdown  of  Risk  Assessment  and  Management  Tools  within  Each  Directorate 
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LM  MIAMI _ 

Phase  1 

Concept 

Exploration 

M  -  11 

11 

II 

M  II 

M  11 

L 

L 

M  -  II 

i  M 

L 

Identify  and  evaluate  risks  and 
impacts  of  system  to: 

i)  evaluate  proposed  concepts 

ii)  define  technical  requirements 

Phase  2 

Demonstration  and 

Validation 

11 

M 

11 

11 

II 

M  -  11 

M 

M  -  11 

11 

M  -  H 

Identify  and  evaluate  technical, 
cost,  supportability  and  schedule 
risks  to: 

i)  evaluate  proposed  systems  ^ 

ii)  define  system  specifications 

Phase  3 

Full  Scale 

Development 

11 

L 

M 

M 

11 

11 

11 

11 

M  -  H 

11 

Identify  and  reduce  all  acquisition 
risks 

Plan  and  control  system  development 
while  reducing  technical,  cost, 
schedule  and  supportability  risk. 

Phase  4 

Production  and 

Deployment 

L 

h 

L 

M 

M 

L 

11 

11 

M 

L 

Plan  and  control  production  and 
deployment  to  reduce  risks. 

Evaluate  technical  and  cost  risks 
when  engineering  changes  are  required. 

Evaluate  program  risk  when  acquiring 
follow-on  quantities. 

Table  3.  2  Usage  Breakdown  and  Classification  of  Tools  for  Risk  Assessment  and  Management  within  the  Program  Office. 


Usage 

H  - 
M-ll  - 

high  amount  of  expected  usage 

moderate  to  high  usage  depending  on  data  availability 

Classification  of 

1  risk  identification 

Breakdown 

M  - 

moderate  usage 

Tools  by  Stage: 

2  -  risk  quantification 

L  - 

little  to  no  usage 

3  -  risk  evaluation 

4  -  risk  management 
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ii  ,  « 


Data  Base 


Mooel  Base 


Data  Base  Mgt.  Svstem 

Model  Base  Mgt.  Svstem 

Provides  appropriate  co¬ 

Provides  separate  routines 

ordination  of  risk  informa¬ 

tailored  to  each  directorates' 

tion  flow  between  the  dir¬ 
ectorates  and  program  man¬ 
ager 

for  each  of  the  phases 

Dialog  Management  System 


Provides  "user  friendly"  input/output 
features 


0  mouse 
0  color  graphics 


Figure  3.1. 

The  "Envisioned"  DSS  for  the  Entire  Weapon  System  Aquisition  Process 
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4.  A  Prototpye  DSS  for  Project  Risk  Management 

A  common  string  that  runs  through  all  weapon  systems  development  and 
procurement  programs  is  the  need  for  project  risk  management.  Project  risk 
management  provides  a  program  manager  with  an'  integrated  package  of 

techniques  that  he  can  use  in  dealing  with  any  unexpected  changes  in  his 

program  plan.  These  techniques  enable  the  program  manager  to  develop  a 
revised  program  plan  by  adjusting  aspects  of  the  scheduling  and  funding  of 
the  full-scale  development,  production,  and  deployment  phases  of  the 
program.  Such  a  plan  must  be  able  to  meet  certain  constraints  and 
objectives  in  an  optimal  manner.  These  tools  also  help  to  quantify  the 

elements  of  risk  associated  with  a  program  and  present  them  in  a  clear  and 

logical  manner,  improving  the  program  manager's  ability  to  deal  with  risk 
as  it  arises  during  a  program.  By  integrating  these  techniques  into  the 
framework  of  a  microcomputer-based  decision  support  system  (DSS),  the 
program  manager  is  provided  with  a  user  friendly,  interactive  mode  for 
using  the  Project  Risk-Management  Module.  Now  that  the  requirements  of 
such  a  system  have  been  touched  upon,  it  is  time  to  identify  the  problem 
that  the  program  manager  faces. 

4 . 1  Problems  and  Risks  in  Program  Planning  and  Management 
The  program  manager  (PM)  is  ultimately  responsible  for  all  technical  and 
business  decisions  associated  with  his  program.  Although  the  PM  usually 
assigns  functional  specialists  to  deal  with  specific  tasks,  it  is  still  his 
responsibility  to  integrate  and  coordinate  their  efforts.  Some  basic  needs 
that  are  essential  to  successful  program  management  include  (Huffman 
[1981]): 

.  a  teamwork  relationship  within  the  program  office 

.  an  in-depth  understanding  of  all  the  program  objectives 
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program  plans  which  are  well  prepared  and  highly  visible 
accurate  and  timely  information  concerning  actual 
progress  or  planned  work 

.  notation  and  evaluation  whenever  the  plan  makes  a  deviation 
from  what  actually  happens 

.  determination  and  implementation  of  corrective  action 
based  upon  trade-off  judgements 
.  follow-up  on  corrective  action 
.  friends,  not  enemies,  for  the  program 
These  needs  along  with  other  requirements  will  be  addressed  later  in  terms 
of  the  Project  Risk-Management  Module. 

A  development  program  can  be  characterized  by  a  combination  of 
interdependent  activities  involving  production  as  well  as  research  and 
development.  Some  of  the  general  activities  present  in  most  weapon  systems 
programs  include  (from  Ingalls,  1984a)  hardware  development,  software 
development,  software  validation  and  verification,  logistics  planning, 
producibility  engineering  planning  (PEP),  facility  start-up,  long-lead 
material  acquisition,  systems  production,  initial  spares  production, 
training  equipment  production,  and  technical  documentation.  Each  of  these 
activities  have  specific  difficulties,  but  some  general  aspects  can  be 
summarized  in  terms  of  scheduling  and  funding  uncertainties.  These 
uncertainties  introduce  risk  into  the  procurement  process.  Other 
complications  can  arise  because  the  program  must  be  accomplished  within  the 
limited  amount  of  funds  available  and  because  certain  of  its  activities 
must  be  done  in  a  prespecified  order.  These  requirements  place  constraints 
on  the  program  and  impede  the  attainment  of  certain  predetermined 
objectives . 

When  a  weapon  systems  program  is  in  the  planning  and  proposal 
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phase  of  development,  planners  must  set  some  schedule  and  cost  estimates 
for  each  activity  involved  with  the  program.  However,  while  these  plans 
are  being  developed,  trade-offs  exist  between  the  need  to  make  the  program 
appealing  and  the  desire  to  cover  all  possible  events.  In  other  words, 
planners  would  like  to  inflate  their  estimates  in  order  to  ensure  that  the 
program  will  come  in  on  time  and  within  the  budget,  but  the  added  funding 
and  time  requirements  may  jeopardize  the  program’s  viability.  Thus,  the 
proposal  that  is  submitted  for  approval  is  usually  a  compilation  of  most 
likely  schedule  and  funding  estimates.  Once  a  program  has  been  approved 

and  a  full-scale  development  has  begun,  many  risk  factors  can  affect  the 

ability  of  the  program  to  meet  its  constraint  levels  and  objectives.  They 
include  risk  of  immature  design,  software  risk,  changes  in  the  production 
rate,  and  inadequacies  in  facility  start-up  funds,  long-lead  funds, 
logistics  support  and  system  availability.  A  summary  (from  Ingalls,  1984a) 
of  the  causes  of  these  factors  and  their  effects  on  a  program  can  be  found 
in  Appendix  C. 

4.2  OBJECTIVES  AND  TOOLS  IN  PROJECT  PLANNING 

Now  that  the  problems  facing  a  PM  have  been  specified,  it  is  time  to 
take  a  look  at  the  management  objectives.  First,  we  must  realize  that  over 
the  course  of  a  program  there  are  many  specific  objectives  related  to 
individual  activities  and  time  frames.  However,  they  can  be  quantified  in 
terms  of  three  general  objectives  which  can  be  minimized.  First,  the 
risk  of  cost  overruns  measured  in  dollars  per  unit  should  be  minimized. 
Second,  the  risk  of  schedule  delays  measured  in  ’’shortages"  should  be 
minimized.  Finally,  the  risk  of  element  failure  measured  in  performance 
levels  of  the  system  should  be  minimized.  The  importance  of  these 
objectives  is  realized  when  a  program  is  being  reviewed  for  further 
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funding.  The  Congress,  as  well  as  the  media,  view  uncontrolled  cost 
growth,  schedule  slippage,  and  low  performance  levels  as  signs  of  program 
failure.  This  may  lead  to  termination  of  the  program,  especially  in  the 
case  of  controversial  programs.  Therefore,  in  order  for  a  program  to  be 
successful,  the  PM  must  be  able  to  keep  the  objectives  under  control. 

Although  the  structure  of  a  program  plan  is  usually  inflexible,  there 
are  certain  procedures  that  the  PM  can  use  to  help  control  his  objective 
values.  These  procedures  include  better  estimation  and  management  methods, 
expansion  of  constraints,  manipulation  of  decision  variables,  and  trade-off 
analysis.  Some  of  these  tools  will  rarely  be  valuable  to  most  programs. 
For  instance,  there  will  be  a  limit  on  better  estimation  and  management 

methods  available  to  a  well-planned  program.  Also,  constraint  expansion 
involves  the  increase  of  available  funds,  which  is  considered  to  be  the 
"last  resort"  approach  to  solving  problems.  This  leaves  the  manipulation  of 
variables  and  trade-off  analysis  as  the  most  promising  tools  for  project 
risk  management. 

Manipulation  of  variables  involves  certain  key  decision  variables  that 
deal  with  the  budgeting  and  scheduling  of  individual  activities.  One  of 
these  tools  involves  stretching  (or  compressing)  the  cost  of  an  activity 
over  time.  Essentially,  this  action  will  determine  the  level  of  intensity 
of  an  activity  and  directly  affect  the  program  funding  requirements  and 
completion  time.  Another  tool  uses  scheduling  variables  and  involves  the 
starting  dates  of  the  activities.  By  postponing  (or  moving  ahead)  the 
starting  dates  of  certain  activities,  the  PM  has  the  ability  to  move 
certain  funding  requirements  into  different  time  periods.  Again,  this 
action  will  help  the  PM  meet  his  constraint  levels  at  the  cost  of  some 
schedule  slippage.  Up  to  this  point,  there  has  been  no  mention  of  the 
performance  objective.  However,  there  is  one  tool  that  affects  the 
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performance  of  a  program  and  it  involves  the  production  rate.  By  changing 
the  production  rate,  the  PM  is  able  to  meet  funding  constraints;  however, 
this  change  can  affect  the  unit  cost  of  each  item  as  well  as  the  ability  of 
the  program  to  meet  production  levels.  For  some  programs,  failure  to  meet 
production  levels  may  affect  the  national  defense  and  degrade  performance 
levels . 

Methodologies  for  trade-off  analysis  allow  decision  makers  involved 
with  a  multiple-criteria  decision-making  problem  to  arrive  at  a  preferred 
solution.  The  PM  may  be  able  to  define  several  feasible  solutions  with 
several  sets  of  objective  values.  There  will  also  be  trade-offs  associated 
with  each  set  of  objective  values  that  describe  how  a  change  in  one 
objective  affects  the  other  objectives.  The  use  of  the  surrogate  worth 
trade-off  (SWT)  method  along  with  this  information  helps  to  quantify  the 
decision  maker’s  preferences.  The  results  obtained  from  this  analysis  will 
be  feasible,  optimal  (if  enough  functional  information  is  known  to  allow 
for  the  use  of  optimization  techniques),  and  preferred,  in  the  eyes  of  the 
PM. 

4.3  CONSTRAINTS  AND  COMPLICATIONS  IN  PROGRAM  PLANNING 

The  previous  set  of  project  risk-management  tools  may  have  seemed  to 
be  all-powerful,  but  some  factors  exist  that  limit  their  effectiveness. 
These  limiting  factors  involve  constraints  and  uncertainties.  First  of 
all,  four  major  constraints  exist  in  most  procurement  problems:  the  amount 
of  funds  available  to  the  program  per  year,  the  total  amount  of  funds 
allocated  to  each  activity,  certain  production  schedules  that  must  be  met, 
and  precedence  relationships  that  may  exist  between  activities  (i.e., 
certain  activities  cannot  start  until  others  are  finished).  The  budget  and 
production  constraints  are  usually  set  when  a  program  has  been  approved. 
However,  the  DOD  is  apt  to  change  them  over  the  course  of  the  program. 
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The  scheduling  constraints  come  about  because  of  the  dependence  that  exists 
between  certain  activities.  Furthermore,  the  presence  of  uncertain  system 
parameters  introduces  risk  into  the  procurement  process.  These  risks  occur 
because  random  variables  are  found  in  the  process.  In  general,  these 
random  variables  represent  the  duration  of  activities,  the  cost  of 
activities,  and  the  quality  or  performance  of  activities  along  with  future 
budget  uncertainties.  Specifically,  they  can  involve  software  errors, 
production  rates,  maintainability,  usability,  and  suppor tabi 1 i ty .  The 
quantification  of  these  random  distributions  in  terms  of  variables  and 
parameters  of  the  program  is  an  important  part  in  the  development  of  a  DSS. 
The  accuracy  with  which  these  and  other  functional  relationships  are 
developed  will  have  a  direct  impact  on  the  acceptance  and  reliability  of 
the  results  generated  during  the  decision  process.  The  handling  of  these 
constraints  and  risks  will  be  detailed  in  the  description  of  our  prototype 
DSS. 

4.4  A  DSS  for  PM 1 s  Problems 

The  PM ' s  need  for  a  DSS  should  be  evident.  His  responsibilities  have 
become  far  too  extensive  and  important  to  be  neglected  in  our  approach. 
The  main  purpose  of  the  program  office  is  to  collect  and  analyze  all 
information  relevant  to  the  program's  management  and  completion.  The  DSS 
we  have  developed  for  this  purpose  addresses  many  of  the  needs  and 
requirements  of  the  PM.  Some  of  them  include 
.  the  ability  to  handle  risk  issues 

the  ability  to  deal  with  multiple  objectives 
the  ability  to  optimize  a  program  plan 
.  the  ability  to  provide  high  visibility  of  the  program  plans 
identification  of  areas  of  concern  and  impending  trouble 
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ease  of  use  by  a  nontechnical  experts. 

Figure  4.1  presents  the  specific  components  we  have  used  in  our  analysis. 

As  described  earlier,  a  DSS  is  comprised  of,  a  data  base,  a  model 
base,  and  a  user  interface.  The  data  base  should  include  objectives, 
constraints,  time-cost  relationships,  and  variable  values.  Functional 
relationships  in  a  generic  form  should  be  present  in  the  data  base  and, 
after  specific  problem  identification,  the  specific  parameters  can  be  input 
into  the  generic  forms  during  network  model  development.  Our  model  base 
includes  the  program  evaluation  and  review  technique  (PERT)  for  network 
analysis,  the  PMRM  for  quantifying  risk,  an  optimization  package  for 
generating  Pareto  optimal  solutions,  the  SWT  method  for  determining  a 
preferred  solution,  and  the  MMIAM  for  impact  analysis.  A  detailed  descrip¬ 
tion  of  each  of  these  components  can  be  found  in  Appendix  D.  .  The  project 
risk-management  module  in  our  system  is  in  effect  a  combination  of  PERT  and 
the  PMRM  used  to  deal  with  development  and  production  activities. 

The  interfacing  techniques  that  are  used  in  our  prototype  take 
advantage  of  the  latest  technologies.  The  use  of  the  ''mouse"  and 
intelligent  screen  layouts  makes  the  DSS  very  user-friendly.  After  initial 
network  modeling,  the  input  routines  require  little  or  no  specialized 
knowledge  of  network  analysis  and  only  basic  knowledge  of  the  program 
activities.  The  output  generated  during  network  and  risk  analysis  is 
presented  as  one  screen  in  the  form  of  a  bar  chart  of  program  activities 
with  attached  risk  and  objective  function  values.  There  are  also 
additional  menues  that  can  be  accessed  in  order  to  look  at  more  detailed 
risk  or  trade-off  information. 

4.5  EXAMPLE  PROBLEM 

In  order  to  develop  our  prototype,  we  had  to  identify  a  specific 
problem  of  the  sort  that  the  PM  must  often  face.  We  opted  for  a  class 
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exercise  from  a  risk  management  workshop  provided  to  us  by  Dr.  Ted  Ingalls 
of  the  Defense  Systems  Management  College  at  Fort  Belvoir,  Virginia.  The 
problem  statement  follows. 

PROBLEM  (Refer  to  Figure  4.2) 

Your  service  is  undergoing  a  budget  cut.  Because  you  have  had  some 
development  test  problems  (although  you  believe  you  have  now  made  the 
necessary  fixes)  and  because  you  are  not  yet  "locked- int o"  a  production 
program,  your  program  is  a  prime  candidate  for  being  cut.  You  must 
identify  the  impacts  of  giving  up  $50M  in  FY  86  (then-year  $).  The  funds 
may  come  from  R  &  D  or  production.  You  also  must  provide  profile  that  will 
give  up  $500M  over  the  next  5  years  of  the  POM.  At  1330  today,  you  must 
make  a  presentation  that  provides  the  following: 

(1)  your  revised  program  plan  (schedule  and  budget)  identifying  where 
the  cuts  will  be  taken 

(2)  the  risks  associated  with  each  element  cut,  along  with  the  risk¬ 
handling  techniques  you  intend  to  use  for  each  element  to 
mitigate  these  risks 

SOLUTION  APPROACH 

The  limited  time  horizon  and  budget  of  this  project  necessarily 
constrained  the  scope  of  the  developed  prototype  DSS.  For  instance,  only 
one  risk  factor--  change  in  production  rate--is  included  in  the  analysis. 
Also,  because  of  a  lack  of  information,  we  were  not  able  to  develop 
suitable  time-cost  relationships  to  allow  us  to  use  optimization  and  the 
SWT  method  in  our  prototype.  Currently,  it  relies  on  the  PMfs  expertise  to 
optimize  the  solution.  However,  the  DSS  will  help  to  identify  feasible 
solutions  with  which  the  PM  can  work.  Specifically,  it  is  aimed  at 
providing  a  revised  program  plan,  due  to  a  budget  cut,  that  takes  risks 
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into  account.  After  the  data  base  has  been  expanded  and  functional 

relationships  developed,  the  DSS  will  be  able  to  identify  Pareto  optimal 
solutions  on  its  own.  Although  the  data  base  is  limited,  the  contents  and 
structure  of  the  model  base  and  interfacing  methods  provide  our  prototype 
with  some  interesting  and  novel  approaches  to  solving  the  project  risk- 
management  problem. 

4 . 6  Analysis  of  Components 

This  section  will  analyze  the  specific  methods,  techniques  and 

functions  that  were  actually  implemented  in  our  prototype  DSS  to  solve  the 
example  problem.  The  prototype  DSS  does  not  include  all  of  the  pertinent 
components,  and  the  functional  relationships  are  somewhat  simplistic. 

However,  it  represents  the  capabilities  and  efficacies  of  such  a  decision 
support  system. 

4.6.1  Network  Analysis 

The  PERT  component  constitutes  the  foundation  upon  which  the  entire 
prototype  is,  built.  The  capabilities  of  PERT  allow  the  organization, 
management,  and  quantification  of  information  needed  in  project  risk- 
management  . 

During  the  development  of  this  component,  we  have  made  some 

adaptations  that  distinguish  our  approach  from  standard  packages.  These 
adaptations  and  general  assumptions  will  be  discussed  in  terms  of  the 
following  basic  steps: 

the  input  of  schedule  and  cost  information 

.  network  modeling 

critical  path  analysis 

.  distribution  of  expected  completion  time 


The  first  criterion  that  PERT  must  satisfy  is  the  ability  to  deal  with 
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the  uncertainties  involved  with  development  and  production  activities. 
PERT  handles  these  uncertainties  by  assuming  a  beta  distribution  of  each 
activity's  duration  time.  In  order  to  develop  these  distributions,  PERT 
requires  that  three  time  estimates  be  obtained.  The  user  must  provide  for 
each  activity  estimates  of 

a  =  the  optimistic  time--execution  goes  extremely  well 
b  =  the  pessimistic  t ime- -every thing  goes  badly 
m  =  the  most  likely  t ime--execut ion  goes  as  expected 
We  also  require  that  planned  budgets  must  be  given  for  each  activity  in  our 
analysis,  and  they  will  be  specified  at  the  same  time  as  a,  b,  and  m.  The 
values  we  use  in  our  specific  example  can  be  found  in  Table  4.1. 

The  first  adaptation  we  made  to  the  PERT  procedure  may  significantly 
improve  the  use  of  network  analysis  techniques.  By  using  a  different 
method  of  sorting,  the  computer  is  able  to  generate  the  network  model  from 
a  set  of  simple  precedence  relationships.  This  makes  our  system  valuable 
to  a  user  with  little  knowledge  of  networking  techniques.  We  have  set  the 
following  precedences  for  our  example, 

HARD  <  PEP,  LLFD ,  FCSU 
SOFT  <  PEP,  SWVV 
PEP  <  INSP ,  TNEQ ,  TDOC 
DTOT  <  INSP,  TNEQ,  TDOC 

This  is  a  significant  advantage  because  the  user  does  not  have  to  develop  a 
network  diagram:  the  computer  can  generate  it.  The  diagram  developed 
for  our  example  can  be  found  in  Figure  4.3. 

The  next  step  in  the  network  analysis  procedure  is  the  calculation  of 
a  critical  path.  The  critical  path  modeling  (CPM)  procedure  identifies  the 
sequence  of  connected  activities  that  require  the  most  time  to  complete.  A 
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detailed  description  of  this  process  can  be  found  in  the  appendix.  This 
path  represents  the  activities  that  the  PM  must  emphasize.  We  have  also 
developed  a  method  for  accounting  for  the  variance  present  in  each  activity 
when  specifiying  the  critical  path.  This  method  produces  a  criticality 
index  for  each  possible  path  a  program  could  follow.  The  PM  can  use  the 

indexes  to  identify  other  activities  that  may  not  be  on  the  critical  path 
but  deserve  to  be  watched  carefully.  The  results  for  our  example  can  be 
found  in  Table  4.2. 

The  final  task  which  the  PERT  component  accomplishes  is  the 
quantification  of  a  probability  density  function  for  program  completion 
time.  The  procedure  for  the  development  of  this  function  is  quite 
straightforward.  The  overall  distribution  is  assumed  to  be  normal,  with  a 
mean  of  P  and  a  standard  deviation  of  a.  The  analytical  probability 
density  function  for  a  normal  distribution  is 

.(x-y)j. 

f  (x)  =  (l/c/nOe  2^2 

In  order  to  quantify  this  function,  we  must  first  identify  the  mean  program 
duration  (  p)  and  its  standard  deviation  (a).  These  values  can  be 
determined  from  the  activities  in  the  critical  path: 

For  our  example 

p  =  57.01  months 

a  =5.76  months 

These  values  are  then  inserted  into  the  probability  density  function,  which 
is  used  along  with  the  duration  times  and  expected-value  concepts  in  the 


PMRM  to  quantify  the  risks. 
4.6.2  Functional  Relationships 
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In  assessing  relationships  for  our  example  problem,  we  were  able  to 
quantify  two  objective  functions: 

f]_(*)  =  cost  overruns  (unit  cost) 
f 2^ * )  =  schedule  delays  ("shortages") 

We  have  already  explained  how  PERT  develops  the  distribution  of  £2  (*),  so 
we  will  now  outline  the  process  we  used  to  develop  the  distribution  of 
f  i(*> . 

First,  we  identified  the  key  variables  that  are  involved  with  the  unit 


cost  as: 


Example  Values 


xi  ~  length  of  facility  start-up  activities 

X2  =  length  of  long-lead  activities 

X3  =  length  of  PEP  activities 

a  ~  production  rate 

E  =  contractor  experience 

D  =  Direct  production  funds  per  unit 


8 

7 

10 

100 

5 

1.91 


Months 


Years 


Million  dollars 

F  =  manufacturers  fixed  costs  100 

We  then  made  the  following  assumptions; 

1)  The  production  rate  (  a )  can  be  expressed  as  a  random  variable 
distributed  normally  with  mean  y  and  standard  deviation  a  • 

2)  y  is  the  planned  production  rate  and  a  is  a  function  of  Xp  X2> 
x  3,  and  E  . 


3)  o(x  p  x  2j  x  3,  E)  =  10/x^  +  IO/X2  +  2OO/X3  +  300/E,  So,  for 
our  example  values,  a  =  16.68. 

4)  The  unit  cost  is  inversely  proportional  to  a.  Therefore,  we 


define 


f  ](*)  =  1.91  +  F/a 


p 
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4.6.3  Partitioned  Mul tiob j ect ive  Risk  ( PMRM)  Analysis 

The  PMRM  provides  our  DSS  with  the  ability  to  quantify  and  represent 
risks  in  a  logical  and  detailed  manner.  Although  we  only  deal  with  one 
random  variable  here,  the  PMRM  can  be  used  for  all  the  areas  of  risk 
present  in  the  procurement  problem.  The  general  steps  that  are  involved 
in  the  PMRM  are  (Asbeck  and  Haimes  [1984]): 

1)  Find  probability  density  functions. 

2)  Partition  the  probability  axis  to  provide  a 
fuller  risk  description. 

3)  Map  the  probability  partitions  onto  the 
objective  value  axis. 

4)  Find  conditional  expectation  values  for 
each  partition. 

5)  Generate  functional  relationships  between 
conditional  expectations  and  policy  choices. 

6)  Use  optimization  and  the  SWT  method  to  generate 
Pareto  optimal  solutions  and  a  preferred  solution. 

In  the  development  of  our  prototype  we  were  not  able  to  complete  steps  5 
and  6.  However,  steps  1  to  4  are  discussed  here  in  the  context  of  both 
objective  functions.  Also,  details  on  the  entire  procedure  can  be  found  in 
the  appendix. 

For  our  example  problem  we  were  able  to  apply  the  PMRM  by  using  an 
analytical  probability  density  function.  As  described  in  the  previous 
sections,  we  used  normal  distributions  for  both  functions.  When  more  risk 
factors  and  random  variables  are  included  in  the  analysis,  random 
generation  and  Monte  Carlo  simulation  techniques  must  be  used  to  develop 


probability  density  functions. 


* 


Partitioning  of  the  probability  axis  is  intended  to  provide  the  user 
with  a  more  complete  view  of  the  distribution  of  risk.  The  partitions 
separate  optimistic,  pessimistic  and  middle-of-the-road  values  and 
represent  them  as  additional  objective  values  to  be  minimized.  In  general, 
if  these  ranges  are  bounded  by  y  +  a,  the  optimistic  and  pessimistic  ranges 
will  each  contain  15.9%  of  the  values  and  the  middle-of-the-road  range  will 
contain  68.27>  of  the  values.  In  particular,  the  completion  time  ranges 
will  be  bounded  as  follows: 

0  to  y- a  will  contain  the  optimistic  completion  times; 
y - a  to  y  +  c  will  contain  the  middle-of-the-road  completion  time; 
y  +  o  to  oo  will  contain  the  pessimistic  completion  time; 
where  y  and  o  are  unique  to  the  current  schedule. 

The  final  step  in  the  PMRM  procedure,  for  our  purposes,  is  the 
calculation  of  the  conditional  expectations  for  each  partition.  First,  we 
have  to  define  the  low,  medium  and  high  value  partitions  as  Dp  D2  »  and 
D  3.  We  now  define  the  expected  value  within  partition  D  as  a  conditional 
expectation 

b 

f  xp(x)dx 
E[X/Di]  =  - 

/  p(x)dx 
a 

where  a  is  the  lower  bound  of  and  b  is  the  upper  bound.  The  function 

p ( x )  is  the  probability  density  funtion  and  x  represents  the  objective 
function  values.  The  conditional  expectations  along  with  the  overall 
expected  value  of  each  objective  can  now  be  presented  to  the  decision  maker 
for  each  alternative.  The  advantage  the  PMRM  has  over  other  risk 
assessment  methods  is  its  ability  to  represent  the  impacts  that  policies 
may  have  on  the  extreme  and  most  probable  cases  along  with  the  overall 


r 
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impact.  The  results  of  our  analysis  can  be  found  in  Figure  4.4. 

4 . 7  Discussion  of  Use 

The  prototype  DSS  we  have  developed  meets  most  of  the  requirements 
that  were  identified  earlier.  In  particular,  the  system  combines  two  very 
powerful  elements,  PERT  and  PMRM,  to  form  the  Project  Risk  Management 
Module.  The  capabilities  that  this  module  provides  to  the  program  office 
include 

the  ability  to  handle  risk 

the  simplif iction  of  replanning  and  rescheduling 
.  high  visibility  of  program  plans 

.  identification  of  areas  of  concern 

.  comparison  between  alternatives 

The  most  important  aspect  of  our  DSS,  however,  is  the  ease  with  which  these 
complicated  components  may  be  used.  The  only  technical  requirement  put  on 
the  user  is  the  ability  to  identify  schedule  and  cost  estimates  for  each 
activity  along  with  the  activities  that  immediately  follow  one  another. 

A  list  of  general  steps  that  will  summarize  the  use  of  our  system 
include  the  following: 

1)  Enter  schedule  and  cost  estimates  along  with 
precedence  relationships  for  each  activity 

2)  Enter  funding  constraints  for  each  of  the 
next  5  years 

3)  Enter  planned  production  rates  for  the  years 
that  involve  production 

The  DSS  will  then  develop  a  network  model,  define  the  criticality  of 
possible  critical  paths,  and  quantify,  partition  and  evaluate  the  objective 
functions  for  an  initial  program  plan.  The  following  information  will  be 
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available  onscsreen: 

*  a  network  model  with  the  critical  path  highlighted 

*  the  next  five  years  of  the  current  program  plan,  represented  as  a 
bar  chart 

*  the  current  objective  values,  displayed  along  the  border  of  the 
bar  chart 

*  any  yearly  funding  constraints  that  have  been  v  iolated- - the se 
will  flash  at  the  bottom  of  the  screen. 

After  examining  the  DSS-supplied  information,  the  user  will  be  in  a 
position  to  proceed  with  the  next  steps: 

4)  Find  the  feasible  plan.  A  "mouse"  will  be  available  to  carry  out 
the  following  options: 

move  an  activity 
.  stretch  an  activity 

.  change  a  production  rate 

.  change  a  constraint 

5)  When  a  feasible  plan  is  found,  recalculate  the  objective  values 
to  reflect  the  risks  involved. 

6)  The  user  can  repeat  steps  4  and  5  to  investigate 
different  options  in  order  to  generate  the  alternatives 
open  to  him. 

7)  The  user  can  then  present  the  different  alternatives 
he  was  able  to  generate  along  with  his  recommendation 
to  DOD  for  approval. 

4.8  Results  and  Extensions 


In  its  current  form,  our  prototype  DSS  can  provide  the  PM  with  many 
valuable  results.  First  of  all,  the  user  interface  is  "friendly"  and  lends 
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itself  to  simple  use  by  non-experts.  Second,  our  system  provides  a  risk 
quantification  method  which  can  deal  with  the  complex  risks  and 
interdependicies  involved  with  weapon  systems  development  and  production. 
Finally,  it  provides  the  PM  with  a  unique  way  of  generating  alternative 
program  plans  along  with  the  assoicated  risks.  However,  this  system  is 
only  a  step  in  the  right  direction.  By  extending  and  improving  upon  this 

approach  our  prototype  will  evolve  into  a  much  more  complete  tool  for 
program  management.  Some  of  the  features  we  foresee  include 

1)  generation  of  Pareto  optimal  program  plans  and 
trade-off  analysis  to  determine  the  preferred  plan 

2)  more  detailed  handling  of  all  areas  of  risk  and 
uncertainty 

3)  impact  analysis  to  determine  the  future  effects 
of  current  trends  and  decisions 

4)  integration  of  the  performance  ojective  in 
the  formulation 

5)  report  generation  and  other  management  aids 

6)  resource  allocation 

Most  of  the  work  associated  with  these  improvements  will  involve  the 
research  and  quantification  of  the  many  functional  relationships  and 
interdependencies  present  in  the  procurement  process. 


Activity 


Name 


Budget  Optimistic 


Most 

Likely 


Pessimistic 


Hardware  Development 

HARD 

50 

9 

14 

23 

DT/OT 

DTOT 

10 

11 

17 

28 

PEP 

PEP 

20 

7 

10 

17 

Software  Development 

SOFT 

40 

9 

13 

22 

Software  V  &  V 

SWVV 

60 

14 

21 

35 

Logistics 

LOGS 

35 

22 

33 

55 

Initial  Spares 

INSP 

200 

ii 

16 

27 

Training  Equipment 

TNEQ 

105 

20 

30 

50 

Tech.  Documentation 

TDOC 

30 

12 

18 

30 

Facility  Start-Up 

FCSU 

100 

5 

8 

13 

Long-Lead  Fund 

LLFD 

40 

5 

7 

12 

Millions  Months  Months  Months 


Table  4.1.  Time  Estimates  for  Activities 
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Activity 

From 

Node 

To 

Node 

Mean 

Variance 

Earliest 

Start  Finish 

Latest 

Start 

Finish 

Total 

Floa  t 

Free 

Float 

Cri t ica 1 

Path 

HARO 

1 

3 

14  67 

S  44 

0 

14 

0 

14 

0 

0 

OTOT 

1 

4 

1?  83 

8  03 

0 

17 

7 

24 

7 

7 

SOFT 

1 

2 

13  83 

4.69 

0 

13 

1 

14 

1 

0 

LOGS 

1 

5 

34  83 

30. 25 

0 

33 

21 

54 

21 

20 

suvv 

2 

5 

22  17 

12.25 

13 

14 

33 

54 

20 

21 

NULL 

2 

3 

0 

0 

13 

46 

14 

14 

1 

1 

FCSU 

3 

6 

8  33 

1  78 

14 

47 

46 

54 

32 

32 

LLFO 

3 

6 

7.  SO 

1  36 

14 

14 

47 

54 

33 

33 

PEP 

3 

4 

10.67 

2.7? 

14 

38 

14 

24 

0 

0 

*  4* 

!NSP 

4 

5 

17.00 

7.11 

24 

24 

i  * 

54 

14 

16 

TNEQ 

4 

5 

31 .67 

25.00 

24 

24 

24 

54 

0 

30 

444 

TDOC 

4 

5 

19  00 

9.00 

24 

36 

36 

54 

12 

18 

NULL 

4 

6 

0 

0 

24 

54 

54 

54 

30 

30 

Path 


Criticality  Index 


C.I. 


HARD,  PEP,  TNEQ 
SOFT,  PEP,  TNEQ 
HARD,  FCSU,  TNEQ 
HARD,  LLFD,  TNEQ 
DTOT,  TNEQ 


1.0 

0.981 

0.965 

0.951 

0.931 


Table  4.2.  Critical  Path  Analysis 
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% 


Flow  Chart  of  Prototpye  DSS 
Figure  4.1 


Now 

March  '85 


M/S  HI 


Hardware  development 


DT/OT 


$  I  0  M 


PEP 

fonivT 


Software 

Pevelopmen 


Software 

V&V 

Paper  Study  /  SIM 

1  /  Test 

$4  M 


$  I  SM  $40M 


Figure  4.2.  Summary  of  Example  Problem 


_ 1 


Logistics 

Planning 


LSA 


LSA  Record 


$20M 


$  1  5M 


Initial  Spares 
$  1  SOM  $50  M 


Training  Equipment 

140M  T  l35M~ 


Facility 

Start-Up 


$ioom 

Long  Lead 


$40l\1 


Tech  Documentation 

$20 

U  $  1  0  M 

Production-1  00 

I 

$  1  9 1  M 

Prod.  500  Unit;: 

$431  M 

Production  Hardware 
Related  Funds 


$1  40M 


0  Funds 


$54  M 


$1  91  M 


$281  M 


$431  M 


$95M 


1500 

$079M 


$fi79M 


,  1500 
$642M 


$04  2  M 


-21 


LOGS 


Figure  4.  3.  Network  Diagram 
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For  Unit  Cost 


E[x/Opt]  =  2.698  Elx/Midl  =  2.922  E[x/Pes]  =  3.265 


Figure  4.4.  Computational  Results  for 
PMRM  Analysis 
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The  Acquisition  Process 

In  general,  the  weapons  systems  acquisition  process  consists  of  an 
initiation  phase  followed  by  four  major  phases:  concept  exploration, 
demonstration  and  validation,  full  scale  development,  and  production  and 
deployment  (see  Figure  A.  1).  A  program  begins  when  one  of  the  major 
commands  identifies  the  need  for  a  weapons  system.  The  process  officially 
begins  when  a  major  command  issues  a  Statement  of  Operational  Need  (SON) 
for  review  by  the  AFSC  and  AFLC,  thus  entering  the  initiation  phase. 

During  the  initiation  phase  the  AFSC  and  AFLC  personnel  assess  the 
technology  and  constraints  which  are  required  to  satisfy  the  needs, 
identify  known  solution  candidates,  and  estimate  necessary  resources  for 
need  satisfaction.  They  send  the  SON  plus  their  comments  to  HQ  USAF  for 
review  and  validation.  If  the  Secretary  of  Defense  (SECDEF)  (or  the  Air 
staff  for  small  programs)  approves  the  program,  the  concept  exploration 
phase  begins.  Although  the  program  has  been  officially  approved,  it  still 
must  compete  with  other  needs  in  the  Planning,  Programming  and  Budgeting 
Office  for  funding.  If  sufficient  funding  is  available,  the  process  of 
identifying  and  evaluating  alternatives  (i.e.  concept  exploration)  actually 
begins . 

To  oversee  the  various  phases--  concept  exploration,  the  acquisition 
process,  and  the  phases  that  follow--a  program  office  (PO)  is  established 
and  a  program  manager  (PM)  is  designated.  Usually,  industrial  contractors 
propose  alternative  solutions  and  Federal  laboratories  or  research  centers 
are  solicited  for  their  ideas  through  a  document  called  the  Request  for 
Proposal  (RFP).  The  RFP  is  structured  to  encourage  competition  and 
innovation.  Its  contents  focus  on  the  operational  needs  that  must  be 
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resolved , 


cost  and  schedule  thresholds, 


operating  environment,  and 


performance  and  logistics  suppor tab i lity  objectives.  After  the  proposals 
are  received,  the  best  alternatives  are  selected  by  a  PM-led  review  team 
on  the  basis  of  four  criteria:  cost,  schedule,  logistics  support ab i li ty , 
and  technical  performance.  During  this  early  phase  of  the  program,  not 
only  should  the  proposals  be  reviewed  using  the  preceding  criteria,  but 
also  the  various  elements  of  uncertainty  in  each  proposal  should  be 
identified  and  an  attempt  made  to  quantify  their  effects  on  the  expected 
cost,  schedule,  logistics  supportab ility ,  and  technical  performance.  In 
this  manner,  the  four  criteria  along  with  their  risks  can  be  utilized  in 
the  decision-making  process.  Such  risk  evaluation  is  critical,  because 
failure  to  recognize  the  possibility  of  cost  overruns  or  schedule  delays 
may  lead  to  serious  problems  in  the  future.  The  impacts  of  these  "risky" 
situations  must  be  evaluated. 

Once  the  be st -candidate  solutions  or  policy  is  chosen,  HQ  USAF 
prepares  a  draft  decision  coordinating  paper  (DCP)  which  documents  the 
results  from  the  concept  exploration  phase.  In  particular,  the  program 
description,  mission  need  revalidation,  goals  and  thresholds,  acquisition 
strategy  summary,  program  alternatives  and  recommendations,  relevant 
issues,  and  risk/impact  analysis  are  contained  in  the  DCP.  The  DCP  goes 
through  a  review  process  (see  Figure  A.l)  and  finally  reaches  the  hands  of 
the  SECDEF.  The  decision  to  proceed  is  officially  documented  by  the 
signing  of  a  separate  decision  memorandum  by  the  SECDEF.  This  initiates 
the  demonstration  and  validation  phase. 

With  the  proposed  solutions  at  hand,  the  demonstration  and  validation 
phase  attempts  to  refine  these  selected  alternatives  through  extensive 
studies  and  analyses,  hardware  development  (as  appropriate),  and  limited 
tests  and  evaluations.  The  objective  is  to  obtain  enough  information  to 
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validate  one  or  more  of  the  selected  solutions  and  to  provide  a  basis  for 
deciding  whether  to  proceed  into  full-scale  development  phase.  The 
emphasis  at  this  point  is  to  try  to  reduce  the  previously  identified 
technical  and  economic  risks  and  uncertainties  surrounding  the  candidate 
solutions  as  well  as  to  reevaluate  the  needs. 

In  order  to  validate  one  or  more  of  the  selected  alternatives,  the 
most  common  approach  is  for  the  PO  to  issue  another  RFP  to  obtain  initial 
system  and  hardware  configuration  specifications,  refined  cost  estimates, 
and  refined  schedule  projections  from  competing  contractors.  After  the 
proposals  are  received,  a  source  selection  board  evaluates  and  then  selects, 
the  best  proposed  systems,  as  in  the  concept  exploration  phase.  The 
competing  prototype  contractors  then  begin  a  prototype  fabrication  phase  to 
further  define  their  projections.  These  prototype  systems  must  allow 
performance  objectives  to  be  evaluated,  but  do  not  have  to  resemble  the 
final  operational  system  in  other  characteristics.  The  PO  then  decides 
whether  more  testing  and/or  further  development  is  needed,  whether  to 
cancel  the  program,  or  whether  to  move  on  to  the  full-scale  development 
(FSD)  phase.  This  decision  is  again  based  on  cost,  performance,  and 
schedule  and  supportability  considerations  and  their  risks.  Each  prototype 
system  must  be  adequately  developed,  so  that  good  estimates  for  the  above 
criteria  and  their  risks  can  be  evaluated. 

At  the  end  of  this  phase,  alternative  solutions  should  have  been 
validated  and  demonstrated;  the  technical,  cost,  supportability  and 
scheduling  risks  should  have  been  identified  and  quantified  as  well.  With 
this  information,  HQ  USAF  prepares  an  updated  DCP  which  is  sent  through  the 
review  process  and  finally  to  the  SECDEF  for  approval.  Official  approval 
by  the  SECDEF  sends  the  program  to  the  full-scale  development  (FSD)  phase. 
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During  the  full-scale  development  phase,  the  system  and,  the  principal 
items  necessary  for  its  support  are  designed,  fabricated,  tested,  and 
evaluated.  The  result  is  to  obtain  a  preproduction  system  that  closely 
resembles  the  final  system  in  order  to  provide  documentation  and  test 
results  on  which  to  base  the  decision  on  whether  it  is  appropriate  to  enter 
into  the  production  and  deployment  phase.  Throughout  the  FSD  phase,  the 
program  office  conducts  many  program  reviews  which  evaluate  how  closely  the 
contractor  is  staying  within  budget,  performance,  and  schedule  constraints 
as  well  as  reevaluating  the  potential  risks  of  cost  overruns,  performance 
deficiencies,  and  schedule  problems  and  their  impacts.  This  results  in 
critical  design  review  (CDR).  At  the  CDR,  the  government  has  its  last 
chance  to  make  (without  significant  additional  costs)  any  amendments  to  the 
system  design  before  it  is  committed  to  hardware. 

The  PO  also  conducts  system  testing  and  evaluation,  which  helps  to 
identify  and  assess  potential  acquisition  risks  so  that  they  can  hopefully 
be  reduced.  Also,  operational  performance  is  evaluated  and  system 
deficiencies  are  identified.  Once  the  system  is  developed  and  the  testing 
is  completed,  the  DCP  evaluation  process  is  again  initiated  and  the 
SECDEF's  approval  initiates  the  production  and  deployment  phase. 

The  objective  for  the  production  and  deployment  phase  is  to 
efficiently  produce  and  deliver  effective  and  supportable  systems  to  the 
operating  units,  but  this  objective  is  seldom  met.  This  phase  includes  the 
production  of  all  system  hardware,  spare  parts,  support  equipment,  data, 
software,  etc.  It  also  includes  verification  of  specification  compliance, 
evaluation  of  production  progress  against  the  planned  strategy,  and  further 
testing  and  evaluation.  Many  unexpected  events,  such  as  Federal  budget 
cuts,  equipment  failure,  changes  in  the  need,  etc.,  can  produce  cost 
overrruns ,  schedule  slippage,  and  reduction  in  the  quality  and  quantity  of 
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items  produced.  In  other  words,  failure  to  adequately  account  for  risks 
and  uncertainty  can  ultimately  lead  to  the  failure  of  the  entire  program  in 
this  final  phase.  It  is  again  very  important  to  identify  the  areas  of 
uncertainty  and  to  evaluate  and  manage  the  risks  in  order  to  develop  a 
flexible  plan  for  production  and  deployment.  Such  risk  assessment  and 
management  must  be  continually  addressed  throughout  this  phase. 
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Figure  A.l:  Details  of  the  weapon  system  acquisition  process 
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APPENDIX  B 

Bi  Risk  Assessment  and  Management  Methods:  A  Review 

The  purpose  of  this  section  is  two-fold:  the  first  objective  is  to 

present  a  general  overview,  philosophy,  and  general  precepts  of  the 

process  of  risk  assessment  and  management.  The  second  objective  is  to 
review  a  collection  of  tools  and  techniques  that  are  available  for 
assessing  and  managing  risks  associated  with  weapon  system  acquisition  from 
both  a  theoretical  and  a  conceptual  viewpoint.  This  section  provides  the 
groundwork  for  Sections  3  and  4  in  the  report,  where  a  number  of  promising 
methods  are  selected  for  a  more  detailed  evaluation  within  the  specific  con¬ 
text  and  needs  of  the  weapon  system  acquisition  process. 

B.  1  The  Process  of  Risk  Assessment  and  Management *  * 

Although  the  study  of  risk  assessment  and  risk  management  (as  both  an 
art  and  a  science)  is  not  new,  its  acceptance  as  a  major  teaching  and 
research  discipline  has  emerged  primarily  in  the  last  decade.  The 
development  of  valuable  theory  (especially  in  the  social  and  behavioral 
sciences  related  to  the  perception  and  acceptance  of  risk),  the  appearance 
of  methodologies  (especially  in  the  mathematical  sciences  and  engineering) 
concerned  with  risk,  and  the  unprecedented  growth  of  public  awareness  of 
the  risks  that  modern  society  must  face  concerning  health,  the  environment, 
and  other  areas--all  these  indicate  a  burgeoning  that  necessitates  and 
justifies  the  discussion  and  careful  execution  of  risk  assessment  and 
management,  particularly  in  large  development  projects  such  as  weapons 
system  development. 


*  This  section  is  adapted  from  "Risk  Assessment  and  Management  in  a 
Multiobjective  Framework,"  Y.Y.  Haimes  and  M.R.  Leach,  in  Decision 
Making  with  Multiple  Objectives,  Y.  Y.  Haimes  and  V.  Chankong  (eds), 
Spr inger- Ver lag ,  Hamburg,  1985  (in  press). 
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Today  risk  assessment  and  management  are  the  domain  of  almost  every 
Federal,  state,  and  local  agency  as  well  as  large  and  small  corporations. 
Most  public  policies  are  formulated  with  explicit  considerations  of  health 
and  safety.  Federal  agencies  such  as  the  U.S.  Environmental  Protection 
Agency,  the  Nuclear  Regulatory  Commission,  the  Federal  Drug 
Administration,  the  Federal  Aviation  Administration,  the  U.S.  Army  Corps  of 
Engineers,  the  Department  of  Defense,  etc.,  have  distinctive  teams  and 
programs  that  address  risk  management  in  their  respective  mandates  and 
areas  of  responsibility. 

In  the  risk  assessment  and  management  process  there  are  a  number  of 
situations  (reflecting  the  degree  of  uncertainty)  that  can  be  encountered 
and  a  number  of  steps  that  need  to  be  performed.  To  avoid  common 
ambiguities  of  terms  and  terminologies,  the  following  definitions  will  be 
used--not  as  universal  definitions,  but  as  a  useful  means  of  communicating 
with  the  reader: 

Risk  si tuations- -si tuat ions  in  which  the  potential  outcome  can  be 
described  by  reasonably  well-known  probability  distributions. 

Imprecision  si tuations--si tuations  having  potential  outcomes  that 
cannot  be  described  in  terms  of  objectively  known  probability 
distributions,  but  which  can  be  estimated  by  subjective 
probabilities . 

Uncertainty  si tuations--si tuat ions  in  which  potential  outcomes  cannot 
be  described  in  terms  of  objectively  known  probability 
distributions . 

Risk  assessment- -a  complete  process  that  encompasses  all  of  the 
following  five  elements  or  steps:  risk  identification,  risk 

quantification,  risk  evaluation,  risk  acceptance  and  aversion, 
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and  risk  management.  The  term  risk  will  be  generally  used 
in  this  paper  to  connote  situations  of  both  risk  and  uncertainty. 

Risk  identif ication--identif ication  of  the  nature,  types,  and  sources 
of  risks  and  uncertainties.  Risk  identification,  as  the  first 
stage  of  risk  assessment,  aims  at  a  complete  description  of  risky 
events  and  elements  of  major  concern  along  with  their  causative 
factors  and  mechanisms. 

Risk  quantification-- formulation  of  appropriate  measures  of  risk  and 
estimation  of  the  likelihood  (i.e.,  probability)  of  occurrence  of 
all  consequences  associated  with  risky  events  as  well  as  the 
magnitude  of  such  consequences. 

Risk  evaluation — selection  of  an  evaluation  procedure  (e.g., 

optimizing  expected  value;  trade-off  analysis)  and  analysis  of 
various  possible  impacts  of  risky  events. 

Risk  acceptance  and  aversion--decision  making  regarding  both  an 
acceptable  level  of  risk  and  its  equitable  distribution.  This 
stage  of  risk  assessment  also  involves  the  development  of  risk 
control  (i.e.,  methods  to  reduce  or  prevent  risk). 

Risk  management- -  formulation  of  policies,  the  development  of  risk- 
control  options  (i.e.,  methods  to  reduce  or  prevent  risk),  and 
the  execution  of  such  policy  options. 

The  last  two  stages  of  the  risk  assessment  process--risk  acceptance  and 
aversion  and  risk  management- -over lap  to  a  large  extent  and  require  the 
subjective  judgment  of  the  appropriate  decision  makers  in  trading  off  the 
noncommens urate  beneficial  and  adverse  consequences  resulting  from  the 
ultimate  "acceptable  risk"  decision.  The  existence  of  these  fundamental 
trade-offs  among  conflicting  and  noncommensurate  multiple  objectives  and 
attributes  demands  the  consideration  of  risk  management  as  an  integral  part 
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of  the  overall  decision-making  process--which  is  the  imperative  premise 
assumed  in  this  report  (Haimes  [1981]). 

Briefly  stated  from  the  perspective  of  multiobjective  decision-making, 
the  risk  assessment  and  management  process  consists  of  two  major  phases 
that  partially  overlap: 

(a)  Information  is  quantitatively  processed  and  evaluated  through 

well-developed  procedures  and  methodologies,  including  the 
quantification  of  risk  and  uncertainty  and  the  development  of 
alternative  policy  options.  The  methodologies  of  risk 

assessment  are  techniques  that  utilize  a  scientific  approach 
toward  estimating  the  probabi 1 ities and  performing  risk  assessment 
(while  excluding  the  explicit  application  of  value  judgments). 

(b)  Value  judgment  is  introduced,  within  the  overall  decision-making 
process,  concerning  what  risks  and  their  associated  trade-offs 
are  acceptable,  what  selections  are  preferred,  what  policies  are 
desirable,  what  constitutes  the  ultimate  decision  (the  best- 
compromise  solution),  and  what  actual  actions  should  be  taken. 

It  is  worthwhile  to  note  that  the  setting  of  value  judgment  is 
critically  important;  it  is  an  integral  part  of  any  decision-making 

process  and  thus  is  integral  to  the  risk  assessment  process  itself.  This 
process  also  serves  as  an  educational  medium  for  the  decision  makers  in 

their  interaction  with  the  analysts;  it  can  help  identify  and  articulate 

the  issues  upon  which  there  is  an  agreement  among  decision  makers  and  also 
those  for  which  there  is  no  agreement;  it  also  helps  to  make  the  implicit 
explicit  (doing  this,  however,  at  the  expense  of  embarassing  decision 
makers  under  certain  circumstances). 

In  many  cases  of  policy  analysis,  a  formal  decision-making  procedure 
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is  appropriate.  Many  of  these  procedures  deal  with  only  one  primary  goal 
or  objective  function.  This  approach  usually  simplifies  the  problem  both 
conceptually  and  computationally,  but  may  prove  to  be  inadequate  when 
decisions  of  great  complexity  are  to  be  made.  An  alternative  approach  is 
to  use  a  decision-making  procedure  that  allows  for  the  consideration  of 
several  objectives  that  may  be  conflicting  and  noncommensurab le . 

Haimes  [1981]  discusses  the  usefulness  of  assessing  risk  within  a 
mu 1 t i ob j e c t i ve  framework.  A  complete  evaluation  of  risks  in  terms  of  their 
trade-offs  with  other  objectives  (for  example,  cost)  necessarily  lends 
itself  to  a  multiobjective  analysis.  Risk  analysis  and  multiobjective 
decision  making  are  further  related  by  the  roles  that  the  analyst  and  the 
decision  maker  play  in  both.  The  role  of  the  analyst  in  multiobjective 
decision  making  is  to  model  the  system,  quantify  objectives,  determine 
trade-offs,  and  assist  the  decision  maker  in  interpreting  the  results. 
Similarly,  in  risk  analysis,  the  analyst's  job  is  to  identify  and  quantify 
risks.  In  each  case,  the  decision  maker  supplies  the  value  judgments  and 
preferences  and,  in  the  case  of  risk  analysis,  determines  the  level  of 
acceptable  risk.  Thus,  the  ultimate  efficacy  of  risk  assessment  lies  in  its 
incorporation  and  utilization  in  the  overall  decision-making  process. 

Two  traditional  measures  of  risk  are  the  expected  value  of  damage  and 
expected  utility.  Kaplan  and  Garrick  [1981]  believe  that  these  approaches 
are  not  adequate.  They  point  out  that  such  scalar  representations  of  risk 
can  be  derived  only  at  the  expense  of  losing  a  great  deal  of  information. 
The  expected-value  approach  also  tends  to  equate  a  low-probability,  high- 
damage  scenario  with  a  high-probability,  low-damage  scenario,  when  these 
situations  are  actually  quite  different  .  McCord  and  de  Neufville  [1982] 
have  demonstrated  empirically  that  there  are  weaknesses  in  the  practical 
use  of  the  expected  utility  theory  in  its  present  form, 
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the  appropriateness  of  expected  utility  decision  analysis  as  currently 
practiced . 

In  many  cases  the  decision  maker  will  be  more  interested  in  the  low- 
frequency,  high-damage  events  than  in  the  overall  probability  disribution. 
Public  perception  of  catastrophic  risks  is  an  important  consideration. 
Fairley  [1981]  states  that  records  of  zero  occurrence  of  catastrophic 
accidents  from  technologically  based  industrial  operations  (such  as  nuclear 
power  generation)  are  often  cited  as  evidence  of  the  safety  of  such 
operations.  He  reaches  the  conclusions  that  such  records  are  actually  of 
little  comfort  in  ruling  out  the  possibility  of  such  catastrophic  events. 
Clearly,  there  is  a  need  for  risk  analysis  methodologies  that  allow  for 
consideration  of  such  risks.  This  capability  can  be  found  in  mul t iob j ec c ive 
approaches,  such  as  the  partitioned  multiobjective  risk  method  ( PMRM) 

(Asbeck  [1982]  or  Asbeck  and  Haimes  [1984]),  the  risk/dispersion  index 
method  (RDIM)  (Rarig  and  Haimes  [1983])  the  multiobjective  statistical 
method  (MSM)  [Haimes  et  al.  [1980]),  the  uncertainty  sensitivity  index 

method  (USIM)  [Haimes  and  Hall  [1977]). 

The  ultimate  goal  of  any  multiobjective  decision-making  methodology  is 
Co  provide  the  decision  maker(s)  with  the  most  useful  information  in  the 
most  effective  manner  and  to  aid  in  arriving  at  a  policy  choice  in  a 
logical  and  consistent  way.  It  is  also  important  not  to  overwhelm  the 
decision  maker(s)  with  too  much  information,  or  else  confusion  may  result. 

A  good  risk  analysis  method,  therefore,  should  express  risks  to  the  extent 
possible,  in  a  clear,  complete,  and  concise  manner.  Information  concerning 
various  impacts  and  trade-offs  is  also  of  great  value.  A  decision-making 
mechanism  should  be  provided  which  can  help  the  decision  maker  arrive  at  a 
policy  based  directly  on  value  judgments  and  preferences. 
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In  addition  to  considering  multiple  objectives,  risk  analysis  can  be 
further  extended  by  considering  explicitly  the  element  of  time.  Simply 
determining  the  probability  of  a  certain  event,  which  is  the  case  in  many 
risk  analysis  techniques,  does  not  give  any  indication  of  short- , medium- , or 
long-term  risks,  or  of  how  risks  change  over  time.  Since  risks  that  may 
not  be  detectable  for  an  extended  period  of  time  can  arise  from 
technological  developments  (for  example,  acid  rain  or  many  carcinogens), 
there  is  a  growing  need  to  develop  risk  assessment  methodologies  that 
explicitly  address  such  long-range  impacts. 

Haimes  [1984a]  considers  this  problem  and  suggests  the  integration  of 
impact  analysis  with  risk  and  uncertainty  assessment  to  form  a  more 
complete  analysis.  Specifically,  the  multistage  multiobjective  impact 
analysis  method  (MMIAM)  (Gomide  [1983]  or  Gomide  and  Haimes  [1984])  could 
be  used.  This  method  explicitly  develops  the  trade-offs  between  various 

objective  functions  at  different  stages  of  the  planning  horizon.  These 
dynamic  trade-offs  are  interpreted  as  the  impacts  that  decisions  made  at 
one  point  in  time  may  have  on  the  state  of  the  system  at  a  future  point  in 
time.  The  quantification  of  risk  impacts  is  an  important  step  toward  a 
more  resilient  risk  and  uncertainty  assessment  and  a  more  robust  decision 
making  process.  Leach  [1984]  and  Leach  and  Haimes  [1985]  further  integrate 
risk  and  impact  analysis  in  a  multiobjective  framework. 

B.2  Selection  and  Classification  of  Methods  for  Preliminary  Review 

We  will  now  select,  classify  and  review  methods  for  assessing  and 
managing  risks  associated  with  weapons  system  acquisition.  There  are 
indeed  a  large  number  of  such  techniques  proposed  in  the  literature  and 

I 

only  some  will  be  selected  for  review  here.  To  help  us  in  our  selection, 
we  perceive  a  particular  weapon  system  acquisition  and  development  as  a 
pro ject--a  customized  one-time  endeavor--designed  to  fulfill  specific  needs 
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of  a  particular  command.  The  major  concern  here  are  the  planning, 
management,  and  control  of  such  a  project.  With  this  perception  and  bias, 
our  selection  and  classification  scheme  and  subsequent  discussions  revolve 
around  two  interrelated  general  aspects  of  weapons  system  acquisiton 
.  risk  assessment  and  management 

.  project  planning,  management  and  control 

For  comprehensiveness,  methods  selected  for  review  are  those  that  attempt 
to  deal  with  the  whole  or  parts  of  either  or  both  of  these  aspects. 
Together  they  represent  a  wide  variety  of  tools  ranging  from  the  very 

specific  (dealing  with  specific  parts  or  aspects  of  risk  assessment  and 

management  or  project  management)  to  the  very  comprehensive  (dealing 
with  both  aspects  comprehensively).  We  believe  it  is  useful  to  start  with 
this  broad  list  in  order  to  give  an  overall  picture  of  what  methods  are 
available.  The  methods  are  then  categorized  into  three  classes  based  on 
their  conceptual  foundations  and  rationale:  (i)  the  heuristic  class,  (ii) 
the  formal  analysis  class,  and  (iii)  the  network  class. 

The  heuristic  class  is  a  mixed  bag  of  heuristic  devices  aimed  at 
specific  parts  of  risk  assessment/management  or  project  management. 
Heuristic  methods  selected  for  review  in  this  report  are  variance  analysis, 
work  breakdown  structure  (WBS)  simulation,  and  the  method  of  moment.  Other  such 
devices  to  be  mentioned  are  Gantt  or  bar  charts,  hierarchical  decision 
modeling, and  other  graphic  aids. 

The  formal  analysis  class  consists  of  those  methods  that  have 
formalized  structure  and  are  predicated  on  some  well-established 
intellectual  roots.  Methods  of  this  sort  selected  for  review  are  mostly 
concerned  with  risk  assessment  and  risk  management  in  a  more  general 


setting  than  simply  project  management  and  control. 


These  include 


deci s ion -mul t ia t t r ibute  utility  theory,  multiobjective  opt imizat ion/ risk 
trade-off  methods,  economics-based  methods,  and  simulation  models. 


The  network  class  is  a  collection  of  network-based  techniques 
primarily  intended  for  project  management,  control,  and  evaluation. 
However,  most  network  techniques  selected  for  review  here  do  have  as  one  of 
their  principal  focal  points  the  explicit  or  implicit  consideration  of  the 
risk  aspects  of  project  management.  These  include  PERT/CPM,  GERT,  SCERT, 
VERT,  WoPAST,  RISCA,  TRACE,  and  MICE,  a  risk  management  model  developed  by 
L.R.  Ireland  and  many  others. 

B;3  Review  Criteria 

To  keep  in  tune  with  the  ultimate  goals  of  this  review  task,  we  shall 
focus  on  the  following  characteristics  of  each  method  under  review. 

1)  Goal.  What  is  the  principle  objective  of  the  model?  Does  it  address 
all  aspects  of  risk  assessment/management  and  of  project  management  and 
control?  If  not,  what  particular  part  (or  parts)  does  it  focus  on?  To  be 
more  specific,  we  shall  probe  to  see  whether  any  or  all  of  the  following  key 
issues  are  addressed  by  each  method  in  the  context  of  project  planning, 
management,  and  control. 

Risk  identification  .  This  involves  identification  of  the  type,  nature, 
and  sources  of  potential  hazards  (damages  or  adverse  effects)  and 
uncertainties.  In  the  context  of  planning,  executing , and  managing  a  large 
scale  project  such  as  weapons  system  acquisition,  four  factors  are  of 
principle  concern- - time ,  cost,  logistic  supportabi li ty ,  and  performance. 
Consequently,  typical  "adverse  effects"  to  be  addressed  in  these  situations 
are  delay  of  the  project,  cost  overrun,  shortage  of  resources  ,  and 
substandard  performance. 

Risk  quantification  .  This  involves  (i)  estimation  of  possible  levels 
(magnitudes)  of  all  potential  adverse  effects  identified  earlier  and  (ii) 
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estimation  of  the  likelihood  (probability)  of  occurrence  of  each  of  those 
levels.  For  example,  we  may  want  to  estimate  how  long  a  project  completion 
data  may  be  delayed  under  various  circumstances  and  what  are  the 
likelihoods  of  such  delays. 

Risk  evaluation /acceptance .  This  involves  identification  of  possible 
impacts  of  various  potential  adverse  effects  as  well  as  determination  of 
acceptable  level  and  distribution  of  risk.  For  example,  before  a  decision 
can  be  made  or  policy  formulated  regarding  delay  or  cost  overrun  of  a 
project,  all  the  significant  impacts  of  such  a  delay  or  cost  overrun  for 
the  overall  program,  the  executing  agent, or  society  as  a  whole  should 

be  first  fully  appreciated. 

Risk  management  .  This  involves  formulation  of  policies,  development 
of  risk  control  measures  (i.e.,  methods  to  reduce  or  prevent  risk),  and  the 
execution  of  such  policy  options. 

2)  Logical  Basis.  Is  the  method  well  structured  and  easily 
comprehensible?  Is  it  justifiable  on  a  theoretical  basis  and/or  an 
empirical  ground?  What  are  the  intellectual  roots,  rationale,  and  the 
underlying  assumptions  upon  which  the  method  is  grounded?  This  should 
provide  a  good  indication  of  how  well  developed  a  method  is. 

3)  Treatment  of  Risk  and  Uncertainty.  Are  risk  and  uncertainty  elements 
treated  explicitly  or  implicitly  (or  not  at  all)?  How  flexible  is  such  a 
treatment  ,  and  is  it  user-friendly?  The  motivation  for  focusing  on  this  issue 
should  be  obvious. 

4)  Supporting  Tools  and  Inputs.  What  basic  tools  does  a  user  need  in 
order  to  apply  the  method?  What  type  and  amount  of  data  and  other  inputs 
are  required?  The  analysis  of  this  issue  should  give  a  good  indication  of 
the  knowledge  base  needed  by  the  user  to  use  the  method.  It  should  also 
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indicate  he  level  of  complexity  flexibility,  and  feasibility  required  by 
the  method . 

5)  Application  Characteristics.  What  are  typical  steps  or  procedures 
used  in  applying  the  method?  What  needs  to  be  done  at  various  steps  and 
for  what  purpose?  This  should  give  an  indication  how  the  method  is 
actually  carried  out  in  typical  applications. 

6)  Application  Experiences .  Has  the  method  been  applied  to  real 
problems?  If  so,  what  types  of  problems?  What  are  the  sizes  and  levels  of 
complexity  of  those  problems? 

7)  Pros  and  cons.  What  are  the  theoretical  and  practical  advantages  and 
disadvantages  of  these  methods?  What  inherent  limitations  does  each  method 
have  ? 

In  what  follows,  we  review  selected  methods  in  the  three  classes 
mentioned  above  using  the  following  format.  First,  a  brief  description  of 
each  method  is  given  which  highlights  the  main  features  listed  above. 
However,  no  attempt  is  made  to  itemize  the  description  into  clear 
subheadings  reflecting  the  aforementioned  features.  Such  features  are 
instead  incorporated  into  tables  at  the  end  of  each  section  (one  for  each 
class)  to  highlight  and  contrast  the  stated  characteristics  of  the  methods 


in  each  class . 
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B. 4  Review  of  Methods  in  the  Heuristic  Class 

Methods  in  this  class  are  composed  mostly  of  heuristics  rather  than 
being  based  on  some  formalized  or  complicated  theoretical  principles.  we 
select  for  review  only  those  methods  that  either  have  application  potential 
or  have  already  been  applied  in  some  real  situations. 

1)  Variance  Analysis  (see,  for  example,  Kerzner,  1979;  Archibald,  1976). 
This  is  a  fairly  well-known  method  designed  primarily  for  measuring 
deviations  of  the  actual  project  cost  and  schedule  from  the  budgeted  cost 
and  the  planned  schedule,  respectively.  In  terms  of  the  ’’goal”  described 
earlier,  the  method  thus  focuses  only  on  specific  parts  of  risk 
quantification  (i.e.,  determining  the  magnitude  of  potentially  adverse 
effects).  Also,  consideration  is  given  only  to  those  parts  of  a  project's 
risk  and  uncertainty  that  are  associated  with  cost  and  time.  There  are  two 
basic  premises  for  considering  both  the  cost  and  time  variation 
simultaneously : 

*  The  cost  variance  compares  deviations  only  from  the  budget  and 
provides  no  measure  for  comparison  between  work  scheduled  and 
work  accomplished. 

*  The  scheduling  variance  provides  a  comparison  between  planned  and 
actual  performance  but  does  not  include  costs. 

Variances  can  be  calculated  in  terms  of  three  basic  variables. 

*  Budgeted  cost  for  work  scheduled  (CBWS)  is  the  budgeted  amount  of 
cost  for  work  scheduled  to  be  accomplished  in  a  given  time 
period . 

*  Budgeted  cost  for  work  performed  (BCWP)  is  the  budged  amount  of 
cost  for  completed  work  plus  that  budgeted  for  level  of  effort  or 
apportioned  effort  activity  completed  within  a  given  time  period. 
This  is  sometimes  referred  to  as  "earned  value." 
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*  Actual  cost  for  work  performed  (ACWP)  is  the  amount  reported  as 
actually  expended  in  completing  the  work  accomplished  within  a 
given  time  period. 

Using  these  definitions,  the  following  variances  are  defined: 
cost  variance  -  BCWP  -  ACWP 
scheduled  variance  =  BCWP  -  BCWS 

Thus,  the  cost  variance  is  the  difference  between  the  budgeted  cost  and  the 
actual  cost  of  a  project  at  a  given  level  of  completion.  The  schedule 
variance  is  a  monetary  measure  that  expresses  the  project  delay  in  terms  of 
budgeted  costs.  The  schedule  variance  can  also  be  expressed  in  terms  of 
time  (hours,  days,  or  weeks)  instead  of  dollars.  These  two  variances  are 
easily  represented  by  plotting  budgeted  and  actual  costs  versus  time,  as 
shown  in  Figure  2.1. 


As  can  be  seen,  the  logical  basis  for  this  tool  is  cost  accounting 
combined  with  common  sense.  The  required  computational  work  is  simple. 
And  the  input  data  required  are  the  budgeted  cost  for  various  stages  of  the 
project,  the  planned  schedule,  and  the  actual  cost  for  various  completed 
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work.  Thus,  good  cost  accounting  reports  are  required  in  using  the  method. 

Although  variance  analysis  does  not  explicitly  treat  the  elements  of 
risk  and  uncertainty,  it  can  be  viewed  as  part  of  risk  quantification 
tools.  It  is,  however,  not  a  complete  risk  quantification  tool  since  there 
is  no  attempt  to  estimate  or  quantify  the  level  of  uncertainty  involved. 
The  method  is  not  useful  for  project  planning.  Rather  it  is  useful  for  on¬ 
line  project  evaluation  and  control.  Cost  and  schedule  variances  can  be 
noted  and  the  sources  of  variance  pinpointed  so  that  the  actual  project 
performance  can  be  brought  closer  to  the  planned  performance. 

In  the  preparation  of  this  report  no  well-documented  practical 
applications  were  found;  however,  according  to  Martin  (1976),  the  method 
has  been  received  with  mixed  feelings  by  various  managers:  some  are  very 
enthusiastic  about  the  method,  while  others  are  most  critical,  believing  it 
is  a  waste  of  time.  Dunne  and  McClary  (1981)  also  reported  similar  resuls 
based  on  responses  of  military  R  &  D  project  managers. 


2)  Method  of  Moments  (McNichols,  1976).  Unlike  variance  analysis,  the 
method  of  moments  focuses  mainly  on  the  probabilistic  nature  of  costs. 
More  precisely,  it  is  designed  primarily  to  estimate  the  probability 
distribution  function  (PDF)  and  the  cumulative  distribution  function  of  the 
total  cost  of  the  project.  Like  variance  analysis,  the  method  of  moments 
can  be  viewed  as  a  tool  for  performing  one  part  of  risk  quantification  task 
(i.e.,  estimating  the  likelihood  of  various  adverse  effects).  Also,  it  only 
addresses  risks  associated  with  cost  elements,  rather  than  dealing  with  all 
four  elements  of  project  management  (cost,  time,  resources,  and 

performance).  In  this  method,  each  cost  component  is  assumed  to  have  a 
known  PDF.  To  compute  an  overall  cost  PDF,  a  form  of  the  required  PDF  is 
assumed,  with  the  parameters  of  distribution  to  be  determined.  The  method 
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proceeds  by  calculating  the  moments  (mean,  variance,  skewness,  etc.)  of  the 
component  PDFs.  Using  these  values  and  the  assumed  type  of  the  overall 
PDF,  the  parameters  of  the  overall  PDF  are  calculated,  thus  determining  the 
required  PDF.  This,  in  turn,  is  used  for  further  risk 
assessment/managment ,  based  on  project  cost. 

The  intellectual  root  for  this  method  is  clearly  predicated  on  the 
theory  of  probability  distribution.  From  the  above  discussion,  all  major 
assumptions  required  should  be  clearly  evident,  the  most  important  being 
the  type  of  overall  PDF.  As  a  supporting  tool,  the  user  should  be  familiar 
with  the  analysis  of  moments  associated  with  probability  distributions. 
Inputs  essential  to  the  method  are  an  understanding  of  how  the  total  cost 
is  broken  down  into  various  components,  sufficient  knowledge  of  the 
probabilistic  nature  of  these  cost  components,  and  some  information  on  the 
probabilistic  nature  of  the  total  cost. 

Formal  practical  applications  of  the  method  have  not  been  found  during 
the  preparation  of  this  report  (perhaps  due  to  the  many  major  assumptions 
required).  The  method  may,  however,  be  useful  when  a  more  detailed, 
rigorous  approach  to  risk  quantification  associated  with  project  cost  is 
impossible  or  impractical. 

3)  Work  Breakdown  Structure  (WBS)  Simulation  (Raiffa,  1968,  and  Military 
Standard ,  1975) . 

This  method  has  exactly  the  same  goal  and  intellectual  roots  as  the 
method  of  moments,  uses  similar  concepts,  and  requires  similar  supporting 
tools  and  inputs.  The  only  basic  difference  is  that,  in  the  WBS 
simulation,  the  form  of  the  PDF  for  the  overall  cost  is  not  assumed. 
Rather,  the  form  as  well  as  associated  parameters  of  the  reqired  PDF  are 
obtained  by  performing  computer  simulation  based  on  the  assumed  PDFs  of  the 
component  costs.  Some  practical  applications  of  the  method  in  the  area  of 
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military  procurement  and  operation  have  been  reported  by  the  Office  of  the 
Chief  of  Naval  Operation  (e.g.,  Jordan  and  Klein,  1975)  and  other  military 
offices  (e.g.,  Directorate  of  Cost  Analysis,  Deputy  of  Comptroller, 
Armament  Division,  Eglin  AFB,  FL,  and  Directorate  of  Cost  and  Management 
Analysis,  Comptroller,  Headquarters,  Air  Force  Systems  Command,  Andrews 
AFB,  DC). 
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4.  Miscellaneous  Tools  for  Cost  Estimations 

Structuring  of  cost  estimates  is  a  very  valuable  and  widely  used 
procedure  in  project  management.  Cost,  together  with  time,  when  presented 
in  the  context  of  a  network  diagram  or  a  work  breakdown  structure,  will 
provide  a  complete  picture  of  the  project  or  program  to  the  program 
manager.  In  general,  cost  estimation  techniques  must  deal  with  costs  at 
several  levels: 

a)  cost  est imates- - f or  tasks  not  yet  started 

b)  budgets-- for  tasks  that  are  in  contract  negotiation  and  planning3 

c)  actual  costs--for  tasks  in  progress  or  completed 

d)  forecasted  costs-to-complete--for  tasks  that  are  in  progress 
and  remaining  tasks  to  be  completed. 

A  number  of  different  aspects  of  cost  analysis  have  been  utilized  to 
help  control  project  costs.  A  brief  description  of  these  concepts  and 
techniques  follows  [adapted  from  Adams,  et  al.,  1978]: 

(i)  Industrial  Engineering  Standards  for  Costing- -In  this  method,  a 
task  is  broken  down  into  its  component  parts.  Whenever 
possible,  standardized  costs  (i.e.,  costs  established  by 
historical  industrial  experience  and  generally  accepted  industry¬ 
wide)  are  assigned  to  the  component  parts.  The  cost  estimate  of 
the  task  is  just  the  sum  of  the  estimates  for  its  component 
parts.  This  approach  is  quite  accurate  for  "state-of-the-art" 
projects  where  little  uncertainty  is  involved, 
ii)  Parametric  or  Statistical  Cos ting- -Parametric  and  statistical 
costing  are  similar  techniques  which  use  basic  technical 

characteristics  of  the  product  to  estimate  the  costs  of  major 
portions  of  the  project  or  the  project  itself.  Statistical 
techniques  are  utilized  to  develop  relationships  for  these 
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technical  characteristics  from  historical  data.  This  technique 
seems  to  be  more  accurate  when  used  on  individual  subassemblies 
or  subsystems,  with  the  costs  then  being  summed  to  develop  the 
project  estimate. 

Parametric  or  statistical  costing  techniques  are  best  used  when 
the  detailed  information  needed  for  engineering  estimates  is  not 
available,  when  accuracy  is  not  particularly  important,  or  when 
speed  and  low  cost  of  developing  the  estimate  is  of  high 
importance . 

iii)  Life  Cycle  Costing  (LCC)--The  life  cycle  cost  of  a  system  is  the 
total  cost  of  acquiring  and  owning  the  system  for  its  entire 
life.  It  includes  the  cost  of  development,  acquisition, 
operation,  support,  and  disposal.  Life  cycle  costing  (LCC) , 
then,  involves  the  consideration  of  life  cycle  costs,  or  segments 
thereof,  in  the  decisions  associted  with  the  development  and 
acquisition  of  the  product.  Thus  the  Hdown~ s tream*  costs  of  the 
project  must  be  estimated  and  considered  by  the  project  manager 
as  his  project  is  designed. 

The  concern  for  life  cycle  costing  originated  with  the  government 
in  the  development  of  weapon  systems.  It  was  found  that  weapons 
were  being  developed  and  procured  with  an  eye  only  to  acquisition 
costs,  schedules,  and  project  performance.  As  a  result, 
decisions  were  being  made  which,  while  holding  acquisition  and 
production  costs  down  and  keeping  performance  up,  were  making  the 
costs  of  ownership — which  costs  include  the  maintenance  and 
operation  of  the  system--prohibitive .  LCC  is  an  attempt  to 
identify  the  down-stream  ownership  costs  associated  with  early 
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design  decisions  and  make  this  added  information  part  of  the 
project  manager's  decision  process. 

iv)  Design -to-Cost-- Design -to- cost  (DTC)  is  a  management  concept 
wherein  rigorous  cost  goals  are  established  during  the  early 
development  of  the  project  effort,  and  the  entire  project  is 
managed  toward  the  achievement  of  these  goals.  This  is 
accomplished  by  practical  trade-offs  between  the  performance, 
schedule,  and  cost  aspects  of  the  project.  Cost,  as  a  key 
project  variable,  is  addressed  on  a  continuing  basis  and  may  take 
precedence  over  both  schedule  and  product  performance 
characteristics . 

Techniques  for  predicting  such  items  as  product  life,  manpower 
costs,  component  reliability,  and  maintenance  requirements  as  a 
function  of  product  design  are  just  beginning  to  emerge. 
Nevertheless,  they  are  being  developed,  and  project  managers  from 
both  government  and  proviate  industry  can  expect  to  see 
increasing  pressure  to  analyze  and  include  life  cycle  costing 
techniques  in  their  project  management  systems, 
v)  The  DARPA  Method--The  method  that  the  Meridian  Corporation  has 
developed  for  DARPA  takes  advantage  of  several  techniques  to  deal 
with  risk  assessment  and  management.  This  approach  is  meant  to 
supplement  traditional  practices  and  aid  the  program  manager  in 
analyzing  and  assessing  the  risk  of  cost  growth  in  order  to 
initiate  preemptive  action.  Basically,  the  method  breaks  risk 
into  three  types--short  term,  mid  term,  and  long  term--and 
concludes  that  no  single  method  exists  that  is  appropriate  for 
all  three.  Several  aspects  from  theoretical  and  empirical  models 
along  with  statistical  theory  are  combined  in  order  to  address 
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all  risk.  The  different  methods  that  were  used  and  the  types  of 

risks  they  address  are  summarized  in  the  following  picture. 

RISK  ASSESSMENT  INDICATORS 
gHggTjnggj  mo  vam  tom  ram 
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___ 
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These  methods  were  integrated  to  form  a  user-friendly  package 
oriented  toward  the  special  needs  of  the  PM.  In  particular,  in 


each  ca 

tegory  of  risk  they  provide 

-  short 

term 

★ 

evaluation  of  earned  value  data 

* 

cost  prediction 

* 

cost  analysis 

-  mid  t 

erm 

* 

cumulative  cost  analysis 

★ 

expenditure  pattern  analysis 

-  long 

term 

★ 

estimate  at  completion 

* 

confidence  limits 

* 

probability  estimates 

This  approach  has  been  used  by  DARPA  and  provides  the  foundation 
for  future  activities  in  risk  management  and  control. 
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In  this  subsection,  some 
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Miscellaneous  Tools  for  Project  Management, 
heuristic  tools  used  primarily  for  project  management,  evaluation  and 
control  will  be  briefly  mentioned.  These  tools,  which  do  not  address  the 
elements  of  risk  and  uncertainty,  include  Gantt  (bar)  charts,  hierarchical 
decision  modeling,  and  miscellaneous  graphic  aids.  ihe  boeing  Method  which 
is  also  discussed  here  does,  however,  address  the  issue  of  risk  assessment 
and  management  explicitly.. 

A  Gantt  chart  (see,  for  example,  Sage,  1977;  Cleland  and  King,  1968) 
is  used  primarily  to  schedule  component  tasks  of  a  project  and  to  monitor 
the  progress  of  each  task.  A  Gantt  chart  is  constructed  by  first  dividing 
a  project  into  component  tasks.  The  estimated  starting  and  ending  time  of 
each  task  is  then  determined.  These  are  plotted  against  time  in  the  form 
of  bar  charts.  The  current  time  and  progress  of  each  task  are  indicated, 
so  it  is  immediately  obvious  which  tasks  are  behind  (or  ahead  of)  schedule 
and  by  how  much.  The  Gantt  chart  is  simple  and  easy  to  understand  and  has 
the  advantage  of  portraying  scheduled  and  actual  progress  simultaneously. 
Gantt  charts  have  some  drawbacks,  however.  The  interdependency  of  the 
tasks  is  not  presented.  Moreover,  such  charts  are  not  useful  for  risk  and 
uncertainty  analysis.  As  can  be  seen,  the  major  project  element  emphasized 
here  is  time  (schedule),  while  other  elements  (cost,  performance,  and 
resources)  are  not  explicitly  considered.  Gantt  charts  are  quite  well 
known  and  have  indeed  been  widely  applied  for  project  monitoring  (see,  for 
example,  Dunn  and  McClary,  1981). 

Other  graphic  aides  (see,  for  example,  Kelly,  1982)  for  project 

monitoring  and  evaluation  include  flow  diagrams,  project  data  sheets,  and 
assessment  charts.  They  all  emphaisze  the  cost  and  schedule  of  the 

projects.  Like  Gantt  charts,  these  graphic  tools  are  based  on  pure  common 
sense.  Their  primary  functions  are  for  bookkeeping  and/or  organization  of 
data  for  visual  inspection  or  for  other  more  complicated  analysis.  These 
project  evaluation  tools  have  also  been  widely  applied  for  project 
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Table  B.l  Summary  of  Main  Features  of  Methods  in  the  Heuristic  Class 
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B.5.1  Decision  Analysis/Multiattribute  Utility  Theory  (DA/MAUT)  (Keeney  and 
Raiffa,  1976) 

The  traditional  decision  analysis  or  Bayesian  approach  to  risk 
assessment  takes  advantage  of  a  PM's  prior  experience  in  order  to  assess 
the  impacts  that  uncertainties  will  have  on  the  cost  of  a  program. 
The  method  uses  the  PM's  subjective  judgments  along  with  results  from 
traditional  estimation  techniques  in  order  to  derive  improved  cost 
estimates.  This  procedure  involves  the  use  of  subjective  probability  and 
random  sampling  in  the  framework  of  Bayesian  analysis  in  order  to  determine 
the  risk  of  exceeding  the  cost  estimate.  The  process  involves  assessing 
the  subjective  "prior"  probability  of  a  primary  event  a  (e.g.,  cost  over¬ 
run,  etc.).  Here,  in  such  a  probability  estimation,  knowledge  of  cross¬ 
impact  analysis  (Gordon  and  Haywoard,  1968;  Gordon,  1969;  Enger,  1970, 

1971,  1972;  Turoff,  1972  and  Sage,  1977)  can  also  be  very  useful.  With 
the  use  of  random  sampling,  the  conditional  probability  of  an  influential 
event  "x"  is  determined,  given  that  the  primary  event  has  occurred  (f(x/a)) 
Normally,  it  is  assumed  that  a  marginal  distribution  of  x,  f(x),  is  assumed 
(usually  normal  distribution)  and  the  posterior  distribution  of  the  primary 
event  "a"  is  computed  using  Bayes'  Theorem. 


f (a)  .  f(x/a) 
f ( a/x)  -  f(x) 

Finally,  Che  prior  and  posterior  distributions  are  combined  with  the  use  of 
a  weighting  factor  to  determine  an  estimate  of  actual  costs.  The  use  of 
this  procedure  is  limited  because  of  the  cost  of  repeated  observations  for 
expensive  systems  and  the  lack  of  accurate  prior  knowledge  for  highly 
advanced  systems. 

The  more  advanced  DA/MAUT  is  a  formal  procedure  to  aid  the  decision 
maker  in  dealing  with  risk  and  uncertainty  in  certain  decision  problems 
where  (a)  the  set  of  alternative  actions  is  small  and  prespecified,  (b)  a 
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set  of 

decision  criteria 

is  to  be  simultaneously  optimized, 

,  and  (c) 

the 

status 

of  environment  is 

uncertain  but 

the  likelihood  of 

occurrence 

of 

each  st 

ate  is  known  or  can 

be  estimated. 

After  a  decision 

problem  as 

been 

properly  defined,  DA/MAUT  entails: 

*  assessing  the  probability  of  each  event  (each  state  of  nature) • 

*  delineating  the  set  of  all  possible  outcomes,  each  given  in  terms 
of  values  of  decision  criteria  in  their  natural  units  (e.g.,  cost 
in  $,  water  pollution  in  mi 1 1 igram/ 1 i ter ,  etc.) 

*  translating  each  of  these  multicomponent  outcome  values  into  a 
single  indexed  value  to  reflect  "preference"  and  attitude  toward 
risk  on  the  part  of  the  decision  maker.  This  translation  normally 
involves  explicit  construction  of  a  multiattribute  utility 
function  that  would  supposedly  represent  the  decision  maker  s 
preference  structure  and  his/her  attitude  toward  risk. 

*  formulating  an  appropriate  decision  criterion  that  will  then  be 

used  in  the  final  selection  of  alternatives.  This  criterion  is 

normally  taken  to  be  expected  utility,  following  the  well-known 
principle  of  maximizing  the  expected  utility  laid  down  over  two 
centuries  ago  by  Bernoulli. 

*  choosing  an  alternative  that  maximizes  the  expected  utility. 

Risk  and  uncertainty  are  incorporated  explicitly  through  construction 

of  a  multiattribute  utility  function  and  through  formulation  of  the 
expected  utility  decision  criterion.  Explicit  analysis  of  risks  cannot  be 
done  conveniently,  however.  The  theoretical  bases  of  this  method  are 
probability  theory,  particularly  Bayes'  theorem,  and  utility  theory.  Many 
key  assumptions  need  to  be  made.  A  set  of  axioms  reflecting  individual 
choice  is  required  to  establish  the  existence  of  a  utility  function  as  a 
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re 


c  _  . e  nrp  f erence .  Some  forms  of  utility  and/or 

presentatalon  of  someone  s  prete 

prefrence  independence  conditions  ace  also  required  to  make  possible  the 

construction  o£  a  multiattribute  utility  function  through  decomposition. 

Finally.  Bernoulli's  principle  is  normally  invoked  to  justify  the  use  o£ 

the  expected  Utility  as  the  ultimate  decision  criterion,  and  the  decision 

problem  simplifies  to  one  of  choosing  an  alternative  action  with 

expected  utility.  The  method  is  ».U  developed  and  has  been  applied  to  a 

wide  range  of  large  scale  problems. 

The  strength  of  the  method,  as  often  claimed  by  its  proponents,  lies 
in  its  attempt  to  guide  a  decision  maker  (or  project  clientele)  to  think 
things  through  and  to  make  value  judgments  along  a  systematic  and  well- 
structured  path  to  bring  about  greater  accuracy,  consistency,  and 
rationality  in  decision  making.  A  full-blown  application  of  the  method, 
however,  demands  a  great  deal  in  terms  of  time  and  effort  both  from  the 
project  clienteles  and  the  analyst.  The  greatest  criticism  of  this  method 
is  directed  toward  the  necessity  of  explicitly  constructing  a 

multiattribute  utility  function.  This  is  a  theoretical  construct  whose 
existence  demands  a  large  number  of  hard-to-verify  axioms  of  individual 
choice.  Indeed,  it  is  hard  to  imagine  that  human  preference  can  be 
represented  by  a  single  equation  (function).  Even  with  its  existence 
verified,  the  actual  construction  process  requires  yet  another  set  of  not- 
so-obvious  •  independence  assumptions  to  allow  construction 

through  decomposition.  The  construction  process  is  often  very  taxing  for 
both  the  decision  maker  and  the  analyst.  After  obtaining  the  utility 
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function,  the  succeeding  steps  (i.e.,  maximizing  expected  utility)  to  reach 
the  final  decision  is  mostly  mechanical ,  with  the  decision  maker's  roles 


replaced  by  his  utility  function. 
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B.5.2  Cost-Benefit  and  Related  Methods  (See,  for  example,  Riggs,  1982). 

Cost-benefit,  cost-effectiveness,  and  risk-benefit  analysis  all 
attempt  to  find  an  alternative  with  the  greatest  economic  efficiency 
measures  by  adding  up  all  the  good  and  bad  consequences  of  each 
alternative.  Such  consequences  are,  in  turn,  measured  in  monetary  terms. 
To  apply  the  methods,  valuation  techniques  are  required  to  express  values 
of  consequences,  good  or  bad,  in  monetary  terms.  Some  valuation  techniques 
are  based  on  a  market  mechanism  where  commodities  and  services  under 
valuation  have  readily  measurable  market  values.  Some  other  indirect 
economic  valuation  methods  rely  on  demand  principles,  shadow  prices,  and 
similar  concepts.  The  intellectual  root  of  these  methods  is  clearly 
economic  theory,  with  the  utilitarian  criterion  (pursuing  economic 
efficiency)  and  the  market  mechanism  serving  as  two  key  underlying 
conceptual  bases. 

Since  1930,  when  the  U.S.  Corps  of  Engineers  first  applied  the  cost- 
benefit  analysis  technique  to  evaluate  watear  resources  projects,  the  method 
and  its  variants  have  been  extensively  applied  to  evaluate  and  appraise 
many  engineering  and  other  development  projects  with  considerable  success. 
For  example,  cost-benefit  analysis  has  recently  been  applied  to  analyze  a 
proposed  runway  extension  at  Kelowna  Airport,  B.C.,  Canada  (Swoveland, 
1981) . 

The  task  of  listing,  estimating,  and  adding  all  consequences  measured 
in  monetary  terms  is  quite  straightforward.  The  method  is  relatively 
simple  to  execute  for  projects  in  which  all  consequences  have  measurable 
market  values.  The  difficulty  and  hence  the  reduction  in  efficiency  comes 
when  trying  to  apply  the  method  to  projects  involving  intangible  or  "soft" 
value s-- such  as  "human  life"  or  "scenic  beauty"-- that  are  very  hard  to 
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price . 


For  projects  in  which  all  alternatives  yield  the  same  benefits  (for 
example,  the  EPA  may  want  to  look  for  different  ways  to  reduce  a  fixed 
amount  of  toxicity  at  a  certain  chemical-waste  dump  site),  cost- 
effectiveness  analysis  is  useful.  The  sticky  task  of  pricing  soft 

factors  (e.g.,  health  effects)  is  removed  in  these  cases  and  the  objective, 
then,  is  to  find  the  alternative  with  the  least  cost. 

The  other  main  drawback  of  these  methods  with  regard  to  the  weapon 
system  acquisition  is  concerned  with  the  fact  that  risk  and  uncertainty 
elements  are  not  easy  to  incorporate  and  to  treat  with  great  flexibility. 

As  a  tool  for  assessing  risks  associated  with  project  management,  the 
emphasis  of  these  methods  is  on  identifying  and  quantifying  the  magnitude 
of  potential  adverse  effects.  Chance  elements  are  viewed  as  a 

characteristic  of  the  environment  that  can  be  taken  into  account  through  the 
market  mechanism  when  performing  the  economic  valuation  of  various 
consequences . 

B.5.3  Multiobjective  Risk  Trade-Off  Methods. 

Most  of  the  techniques  discussed  thus  far  embed  the  elements  of  risk 
and  uncertainty  in  some  other  elements.  (For  instance,  in  DA/MAUT ,  risk 
and  uncertainty  elements  are  embedded  in  utility  functions  and  expected 
utility,  and  in  cost-benefit  analysis,  risk  and  uncertainty  are  treated 

as  part  of  the  environment  and  reflected  in  the  monetary  values  of  various 
consequences  through  the  valuation  process.  However,  the  class  of 
multiobjective  risk  trade-off  methods  discussed  here  formulate  explicit 
measure  of  risks  and  include  them  as  additional  decision  criteria  to  be 
optimized  simultaneously  with  original  decision  criteria.  This  allows  the 
decision  maker  to  consider  risk  explicitly  on  its  own  merit.  Trade-off 
analysis  between  risk  and  other  decision  criteria  can 


be  carried  out 
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convenient ly . 

The  ultimate  objective  in  this  case,  then, is  to  assess  each 
alternative  action  based  on  original  and  new  risk-related  decision 
criteria . 

Multiobjective  risk  assessment  methods  address  two  specific  issues: 

(a)  How  should  risk-related  performance  indices  be  constructed  to 
reflect  the  concerns  of  all  concerned? 

(b)  Given  such  risk  indices,  how  should  risk  be  evaluated  against  all 
other  decision  criteria?  Successfully  answering  these  issues 
leads  to  trade-off  analysis. 

Four  tools  have  been  developed  at  Case  Western  Reserve  University  to 
deal  with  issue  (a). 

( i)  The  Partitioned  Multiob  jective  Risk  Method  ( PMRM)  (Asbeck  and  Haimes^ 
1983) 

This  method  provides  a  procedure  for  formulating  risk-related 
performance  indices  in  projects  where  extreme  events  are  of  major  concern. 
Extreme  events  are  defined  as  events  that  have  a  low  probability  of 
occurrence  but  have  very  devastating  effects  should  they  occur. 
Formulation  of  risk-related  index  through  the  traditional  "expected  value" 
is  inappropriae  in  this  case  since  extreme  events  are  not  highlighted.  The 
PMRM  proposes  to  alleviate  this  drawback  by  partitioning  the  "damage  scale 
into  various  sections,  as  shown  in  Figure  B.  2. 
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A  risk  index  reflecting  extreme  events  is  then  formulated  as  the 

normalized  expected  damage  of  extreme  events  (given  by  f°°  sf(s)ds/9  . 

sk  p 

This  risk  index  is  then  incorporated  into  the  original  problem  as  an 
additional  objective  function.  Subsequent  trade-off  analysis  can  then  be 
performed.  A  more  detailed  descript  ion  of  this  procedure  is  part  of  the 
Appendix. 

(ii)  The  Uncertainty  Sensi t ivi ty  Index  Method  CUSIM)  (Haimes  and  Hall, 

1977)  . 

This  method  provides  a  procedure  for  formulating  a  sensitivity  index 

reflecting  the  uncertainty  of  our  knowledge  on  key  project  parameters.  If 

project  management  decisions  are  sensitive  to  key  project  parameters  and  if 

there  is  imperfection  in  our  knowledge  of  the  values  of  those  parameters, 

explicit  consideration  of  such  imperfection  through  use  of  a  sensitivity 

index  seems  quite  appropriate.  If,  for  example,  a  decision  criterion  is  a 

function  of  decision  variables  xj_  ,...,xn  ,  given  that  key  project 

parameters  are  ,  .  .  .  ,otm(  i  .  ef=f  (x]_ ,  .  .  .  ,  xn  : cx  i ,  .  .  .  , cxm ) ) ,  then  a  sensitivity  index  of 

f  with  respect  to  parameter  a j_  evaluated  as  a  certain  decision  po.it 

x^  ,  x*  can  be  formulated  as  [ _ L  [  n  Ll  >  ^  1  .  Each  of  these 

i 

sensitivity  indices  can  then  be  appended  to  the  original  set  of  decision 
criteria  for  further  trade-off  analysis. 

( ii i )  The  Risk/Dispersion  Index  Method  ( RD IM )  ( Rarig  and  Haime s ,  1983 
This  approach  provides  an  alternative  formulation  of  sensitivity  index 

to  the  USIM.  The  heart  of  the  RDIM  is  the  construction  of  a  sensitivity 
measure  that  accounts  for  the  effects  of  variations  in  the  nominal  values 
of  the  random  variables  a  .  It  is  assumed  that  ocj  '  s  are  independent 
random  variables  with  known  finite  means  and  variances.  The  sensitivity 
measure  ft  ,  called  the  dispersion  index,  which  is  interpreted  as  a  first- 
order  approximation  the  the  standard  deviation,  is  then  incorporated  in  a 
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multiobjective  optimization  formulation.  The  dispersion  index  can  also  be 
interpreted  a  measure  of  the  size  of  the  neighborhood  about  the  nominal 
optimal  solution  in  which  the  actual  solution  is  most  likely  to  occur.  The 
method  also  derives  a  sensitivity  trade-off  (when  using  the  e  -constraint 
formulation  with  the  surrogate  worth  trade-off  (SWT)  method),  which  gives 
an  explicit  representation  of  the  trade-offs  between  the  sensitivity 
measure  ft  and  the  other  objective  functions.  Since  the  RDIM  incorporates 
the  SWT  method,  it  generates  all  needed  Pareto  optimal  solutions  to  the 
multiobjective  risk  problem  (see  Rarig  and  Haimes,  1983). 

The  dispersion  index  is  particularly  useful  in  decision  making.  The 
information  that  ft  conveys  to  the  decision  maker(s)  can  be  readily 
understood:  the  larger  the  value  of  ft,  the  greater  the  possibility  that  he 
actual  solution  will  deviate  significantly  from  the  nominal  solution. 
Since  ft  is  a  scalar- va lued  quantity  and  is  independent  of  the  number  of 
objectives,  any  decision  maker  who  desires  to  minimize  ft  will  not  be 
confused  by  a  deluge  of  sensitivity  information  that  needs  to  be  analyzed 
at  each  prospective  solution  point  (alternative  policy  option). 

Note  that  the  fundamental  difference  between  the  RDIM  and  the  USIM  is 
that  the  former  generates  a  sensitivity  index  on  the  basis  of  probability 
distributions  (conditions  under  risk),  whereas  the  latter  generates  a 
sensitivity  index  without  any  reference  to  probability  distributions 
(conditions  under  uncertainty). 

( iv)  The  Mul tiob  jective  Statistical  Method  (MSM)  (Haimes  e t  al . ,  1980  ) 

This  method  provides  a  general  framework  for  modeling  multiobjective 
problems  with  inherent  stochastic  elements  that  may  be  due  to  either 
probabilistic  inputs  or  the  inherent  stochastic  structure  of  the  system. 
The  MSM  was  developed  for  the  U.S.  Army  corps  of  Engineers  to  account  for 
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the  risk  of  flooding  in  the  design  and  management  of  interior  drainage 
systems.  The  method  is  an  integration  of  multiobjective  optimization  (the 
SWT  method)  and  statistical  simulation  models  ( Stanford- type  stream  flow 
simulation  models)  to  assess  the  probability  of  risk  events  and  their 
consequences.  The  risk  functions  in  the  MSM  are  first  constructed  as 
functions  of  two  state  variab les- -pond  duration  and  pond  evaluation  of 
interior  floodings.  These  to  state  variables  are  then  related  to  the 
system's  decision  variables,  x,  using  Stanford- type  stream  flow  simulation 
models.  Historical  records  associated  with  two  random  variables-- 
precipitation  and  stream  flow--are  then  used  to  generate  conditional  and 
joint  probabilities  (as  appropriate)  for  the  ultimate  development  of  the 
expected  value  of  the  appropriate  risk  functions.  The  set  of  ordered  pairs 
of  the  expected  value  of  the  jth  risk  function,  fj(*k),j  «  1,  2,  ...,  J, 
and  its  associated  policy  decision  (xk)  for  k  =  1,  2,  ...K  is  used  to 
generate  the  needed  functional  relationship  fj(x)  through  a  regression 
analysis  technique.  The  completion  of  this  last  step  yields  to 
quantifiable  risk  functions  amenable  to  optimization  via  the  SWT  method, 
where  Pareto  optimal  policies  and  their  associated  trade-offs  are  generated 

i 

as  part  of  the  risk  assessment  process. 

To  deal  with  issue  (b)--how  to  evaluate  risk  against  other  decision 
criteria--a  number  of  techniques  are  available  to  assist  a  decision  maker 
to  deal  with  decision  problems  under  multiple  decision  criteria  (see 
Chankong  and  Haimes,  1983).  The  multiattribute  utility  function  approach 
described  earlier  is  one  of  the  more  commonly  used  techniques.  It  deals 
with  the  multiplicity  of  decision  criteria  through  the  utility  function. 
Another  class  of  techniques  relies  on  appropriately  designed  weighting 
schemes.  Here  we  would  like  to  focus  on  one  class  of  multiobjective 
analysis  tools  that  relies  on  trade-off  analysis.  A  technique  in  this 
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class  and  its  variants  developed  at  Case  Western  Reserve  University  are  the 
surrogate  worth  trade-off  (SWT)  method  (Haimes  and  Hall,  1974),  the 
interactive  SWT  (ISWT)  method  (Chankong  and  Haimes,  1978),  and  the 
multiobjective  multistage  impact  analysis  method  (MMIAM)  (Gomide  and 
Haimes,  1983).  The  specific  goal  of  all  these  methods  is  to  help  the 
decision  maker  choose  the  best-compromise  or  most-preferred  alternative  by 
trading-off  among  various  decision  criteria.  The  inte 1 lectural  roots  of 
these  methods  are  mathematical  programming,  the  concept  of  Pareto 
optimality  and  trade-off  analysis,  and  the  concept  of  the  marginal  rate  of 
substitution.  The  methods  in  their  present  versions  are  designed  for 
multiobjective  decision  problems  that  can  be  represented  by  continuous 
mathematical  programming  models.  In  addition  to  standard  abstract 
assumptions  required  in  order  to  use  existing  optimization  techniques,  a 
minor  assumption  regarding  individual  choice  (namely,  that  a  person's 
indifference  band  of  preference  exists)  is  assumed. 

Typical  steps  in  applying  these  methods  after  an  appropriate  model  has 
been  constructed  are  (i)  generate  Pareto  optimal  (noninferior,  efficient, 
nondominated )  alternatives,  (ii)  obtain  appropriate  trade-off  information 
associated  with  each  generated  Pareto  optimal  alternative,  and  (iii) 
interact  with  the  decision  maker  to  solicit  his/her  preference  for  those 
trade-offs.  Added  efficiency,  usability,  and  flexibility  are  envisioned  if 
these  steps  are  executed  interactively  and  on-line  as  part  of  a  well- 
designed  decision  support  system.  Since  the  SWT  method  forms  a  core  of  all 
of  the  above  procedure,  we  describe  it  briefly  below  and  in  more  detail  in 
the  Appendix. 

(v)  The  Surrogate  W  o  r  t h  T  r ade-0  f  f  Method 

This  method  recognizes  that  optimization  theory  is  usually  much  more 
concerned  with  the  relative  value  of  additional  increments  of  the  various 


noncommensurab le  objectives,  at  a  given  value  of  each  objective  function, 
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than  it  is  with  their  absolute  values.  Furthermore,  given  any  current  set 
of  objective  levels  attained,  it  is  often  much  easier  to  turn  to  decision¬ 
makers  (DMs )  to  assess  the  relative  value  of  the  trade-off  of  marginal 
increases  and  decreases  between  any  two  objectives  than  it  is  to  assess 
their  absolute  average  values.  In  addition,  the  optimization  procedure  can 
be  developed  so  it  is  no  more  than  assessing  whether  one  more  quantity  of 
one  objective  is  worth  more  or  less  than  that  lost  by  another  at  any  given 
level  of  each  of  them.  An  ordinal  approach  can  then  be  used  with  much 
less  concern  for  the  distortions  that  relative  evaluation  introduces  into 
attempts  to  commensurate  the  total  value  of  all  objectives. 

A  detailed  discussion  of  the  SWT  method  is  available  in  the  Appendix 
and  therefore  only  a  brief  summary  of  it  is  presented  here: 

(i)  The  SWT  method  is  capable  of  generating  all  needed 
noninferior  solutions  to  a  vector  optimization 
prob lem . 

(ii)  The  method  generates  the  trade-offs  between  any  two 

objective  functions  on  the  basis  of  duality  theory  in 

nonlinear  programming.  The  trade-off  function  between 

the  f  and  f;  objective  functions,  Xji  ,  is  explicitly 

i  J  J 

evaluated  and  is  equivalent  to 

3 f * ) 


(iii)  The  decision  maker  interacts  with  the  systems  analyst  and  the 
mathematical  model  at  a  general  and  very  moderate  level.  This  is 
done  via  the  genration  of  the  surrogate  worth  functions,  which 
related  the  decision  maker's  preferences  to  the  noninferior 
solutions  through  the  trade-off  functions.  These  preferences  are 
the  objective  function  space  (more  familiar  and 


constructed  in 
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meaningful  to  decision  makers)  and  only  then  transferred  to  the 
decision  space.  This  is  particularly  important,  since  the 
dimensionality  of  the  objective  function  space  is  often  smaller 
than  that  of  the  decision  space.  These  prefrences  yield  an 
indifference  band  where  the  decision  maker  is  indifferent  to  any 
further  trade-off  among  the  objectives. 

(iv)  The  SWT  method  provides  for  the  quantitative  analysis  of 
noncommensurable  objective  functions. 

(v)  The  method  is  well  suited  for  the  analysis  and  optimization  of 
multiobjective  functions  that  involve  multiple  decision  makers, 

(vi)  The  method  has  an  appreciable  computational  advantage  over  all 
other  existing  methods  when  the  number  of  objective  functions  is 
three  or  more. 

The  SWT  method  and  its  extensions  have  been  extensively  applied  to 
large  scale  problems  such  as  water  resources  (e.g.,  the  Maumee  River  Basis 
Planning  project,  Haimes,  1981,  Haimes  et  al.,  1979;  Das  and  Haimes,  1980) 
and  energy  storage  systems  (Chankong  et  al.,  1981;  Tarvainen  and  Haimes, 
1981) . 

Returning  to  the  problem  of  assessing  and  managing  risks  associated 
with  the  management  of  large  projects,  both  issues  (a)  and  (b)  must  both  be 
addressed.  A  multiobjective  risk  assessment  method  must  thus  combine  a 
suitable  method  (e.g.,  USIM,  RDIM ,  PMRM,  or  MSM)  for  dealing  with  issue  (a) 
with  one  (e.g.,  SWT,  ISWT,  or  MMIAM)  that  deals  with  issue  (b).  The 
following  steps  are  commonly  taken  in  a  typical  appication  of  such 
multiobjective  risk  assessment  methods: 

1)  Develop  a  mathematical  programming  model  representing 
the  decision  problem  of  interest. 
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2)  Formulate  suitable  risk- re lated  indices  using  the 
appropriate  procedures  described  here  earlier. 

3)  Append  these  risk-related  indices  to  the  original 
mode  1 . 

4)  Generate  Pareto  optimal  alternatives  as  well  as 
associated  trade-off  information  between  risk- 
related  objectives  and  other  objectives. 

5)  Interact  with  the  decision  maker  to  arrive  at 
a  final  decision. 

It  can  be  seen  that  this  type  of  procedure,  the  treatment  or  risk  and 
uncertainty  can  be  quite  explicit  and  flexible. 

B . 5 . 4  Simulation  Models 

General  simulation  models  can  be  very  useful  tools  for  risk  assessment 
and  management.  A  typical  example  of  such  models  is  that  developed  by  the 
General  Systems  Department  of  the  University  of  Southern  California,  which 
applies  systems  dynamics  simulation  to  the  problem  of  managing  delays  and 
descriptions  (D  &  D).  This  approach  helps  to  manage  the  risk  involved  with 
D  &  D  by  linking  their  impacts  to  cost,  schedule,  and  performance  goals. 
The  approach  uses  systems  dynamics  simulation  to  model  the  complex 
relationships  involved  with  a  procurement  problem.  The  general  steps 
involved  with  systems  dynamics  include: 

1)  problem  statement 

2)  identification  of  key  factors  and  causal  relationships 
involved  in  the  program 

3)  modeling  of  complex  relationships  as  chains 
of  simpler  relationships 

4)  reformulation  of  the  model  as  a  flow  diagram 
and  representation  of  it  as  a  set  of  difference 
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equations 

5)  estimation  of  parameters  of  the  model 

6)  tesing  the  model  to  increase  confidence 

7)  simulation  of  the  effects  of  proposed 
changes  (delays  and  disruptions) 

8)  use  of  trade-off  analysis  to  determine 
the  best  alternative 

This  approach  has  been  applied  ,  with  success,  to  a  problem  faced  bv 
Ingalls  Shipbuilding  Inc.,  a  division  of  Litton  Industries,  in  1976.  It 
proved  to  be  an  objective  tool  for  the  representation  of  D  D.  However, 
systems  dynamics  is  not  widely  taught  and  it  is  therefore  not  a  widely 
accepted  technique.  Therefore,  this  will  not  be  further  considered.  Table 
B.2  summarizes  the  essential  features  of  methods  in  the  "formal  analysis"  class 
described  in  tills  section. 

B . 6  Review  of  Methods  in  the  Network  Class 

The  primary  focus  of  the  methods  reviewed  in  the  last  section  is  risk 

assessment  and  management  while  the  project  management  aspect  plays  a 

secondary  role,  if  at  all.  The  methods  reviewed  in  this  section,  on  the 

other  hand,  have  project  management  as  the  nucleus  around  which  other 

features,  such  as  risk  analysis,  are  built.  This  is  hardly  surprising 

since  they  are  developed  by  the  people  whose  primary  concerns  are  project 

planning,  management,  monitoring,  evaluation,  and  control.  As  their 

experience  accumulates,  the  need  to  explicitly  consider  the  elements  of 

risk  and  uncertainty  associated  with  managing  large  projects  becomes 

obvious.  This  brings  about  considerable  efforts  in  developing  risk 

assessment  and  management  capabilities  and  adding  them  to  the  existing 

powerful  project  management  network-based  tools.  These  tools  have  gained 

increasing  use  because  of  the  ease  with  which  problems  can  be  modeled  in 


Table  B.2.  Summary  of  Main  Features  of  Methods  in  the  Formal  Analysis  Class 


Decision  Analysis/Muiliatlribute 

Utility  Theory 

Cosl -Benefit  Analysis 
and  Variants 

Mult lobjec live  Risk  Trade-Off 

Methods 

Goal 

•  To  find  an  alternative  with  maximum 
espected  utility  representing  aggregate 
preferences. 

•  To  find  an  alternative  with  greatest 

economic  efficiency  (net  benefit  or 
ex  pec  led  net  benefit). 

•  To  find  the  best -compromise  alternative  which 

le  also  Pareto  optimal  through  risk  trade  off 
analysis. 

Areas  of 
Emphasis 

(with  respect  to 

*  All  aspects  of  risk  assessment  In 
general  decision  making  situation 

•  Possible  use  for  project  evaluations. 

*  Project  appraisal  and  evaluation. 

*  Risk  quantification  and  evaluation  In  general 

decision  making  situation 

IsOglcai 

Basis  A 

Intellectual 

Koota 

•  Probability  laws 

•  Utility  tlieory  (axioms  of 

Individual  choice,  Bernoulli's 

Principle) 

•  Economic  theory  (utilitarian 
economic  efficiency) 

*  Market  mechenslm  as  a  means  of 
aggregating  risk  and  multiple 
criteria 

•  Nonlinear  optimization  theory 

•  Probability  distribution  theory 

•  Pareto  optimal  concspte 

•  Marginal  rata  of  substitution  concept 

Assumptions 

•  Utility  and/or  preference 

Independence  conditions 

*  Preference  can  always  be  measured 

In  monetary  terma 

*  Usual  assumptions  to  use  op  tl  max  at  Ion  techniques 

•  Indifference  trade  off  band  of  an  Individual  exists 

Support  knf 
Tools  and 

Input  Requirt- 
manta 

•  Techniques  for  constructing  multi- 
attribute  utility  function 

•  Mathematical  expectation 

*  Data  requirement  Is  moderate 

*  Effort  requirement  la  astansive 

*  Valuation  techniques  to  express 
values  of  consequences  In  monetary 
terms 

*  Cost  data  required  is  estenslva 

*  Efforts  required  ara  moderata 

•  Opt  1  ml  cat  Ion  techniques 

♦  Trade-off  analysts 

♦  Mathematical  expectation 

•  Data  and  afforia  required  may  be  axtenslve 

Treatment  of 
Risk  and 
Uncertainty 

•  Risk  and  uncertainty  are  Incorporated 
explicitly  in  the  form  of  utility 
function  and  espected  value 

•  No  explicit  risk  analysis  provided 

•  Chanca  element  viewed  as  char- 

act  eristics  of  environment  and  aggre¬ 
gated  through  market  mechanism 

*  Limited  uae  of  objective  probabilities 

•  Rlek  Is  quantified  and  dealth  with  explicitly 

through  Irada-off  analysis 

Application 

Characteristics 

•  Identify  set  of  alternatives,  set  of 
states  of  nature  and  set  of  decision 
criteria 

•  Identify  values  of  all  possible  outcomes 

•  Construct  mull  la  1  tribute  utility  function 

•  Formulate  "expected  utility*  and 
maximize 

•  List  all  consequences  of  a  project 

•  Uae  valuation  techniques  to  express 
these  consequences  (good  or  bad)  lr» 
monetary  terms 

•  Add  all  good  and  all  bad  consequences 
separately  using  discount  factor 

•  Compare  the  two  In  some  appropriate 
way  and  make  decision 

•  Develop  appropriate  model 

•  Formulate  suitable  risk  raUled  Indices  using 
procedures  described 

•  Append  risk  Indices  to  original  model 

•  Generals  Pareto  optimal  allamalivaa  and 

asaoclalad  risk  trade  offs  j 

•  Interact  with  DM  to  arrive  al  a  final  daclslon 

Application 

Experience 

•  Numerous  In  both  large  and  small 
problems  (e  g.,  Hospital  Blood  Bank 
(Keeney,  1972),  Air  Pollution  In  New 

York,  (Ellis  and  Keeney,  1972),  Fire 
Department  Operation  (Keeney,  1973), 
Airport  Development  for  Mexico  City 
(de  Neufvllle  and  Keeney,  1972) 

*  Numerous  particularly  In  water 

resource  and  olher  Urge  engineering 
or  development  projects 

•  Maumea  Rlber  Basin  Planning 

•  Energy  Storage  System 

•  Acid  rain 

Pros  A  Cons 

•  Systematic  and  well  structured  guide 
to  making  value  Jut%menls 

•  Demanding  on  users 

•  Relies  too  lieavily  on  the  abalracf 
utility  function,  the  existence  and  con 
•truclton  of  which  rely  on  herd- to- vartf; 
abstract  assumptions. 

•  Not  all  consequences  can  be  "priced* 
(some  valuation  Issues  are  still 
controversial) 

•  Not  really  effective  as  risk  assess¬ 
ment  tool. 

1 

•  Require  mathematical  programming 
formula  lion-can  be  quite  eetenslve  end 
dele  demanding 

•  Risk  measures  are  formulated  and  treated 
expllctjjjf— great  fleelbllHy 

•  Wtion  It  appttee,  provhloa  a  ayatomatlc  and 
structured  guide  for  DM  to  make  more  Infori.wd 
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network  fonn  and  because  of  the  ability  to  model  complex 
compounding  simple  systems.  In  what  follows,  we  review 
starting  with  the  standard  CPM/PERT  for  project  management. 


systems  by 
these  tools, 
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1)  Critical  Path  Method  (CPM)  and  Project  Evaluation  Review  Technique 
(PERT)  Moder  et  al.,  1983. 

The  CPM  and  the  PERT  are  two  closely  related  project  management 
techniques  that  are  based  on  network  analysis.  They  both  focus  on  the  time 
element  (scheduling),  although  consideration  of  cost  and  resources  may  be 
brought  about  in  the  postanalysis  stage.  Their  primary  purpose  is  to  help 
management  determine  how  time  delays  in  certain  components  will  affect 
overall  progress  where  slack  time  is  available  and  what  elements  are 
crucial  to  remaining  on  schedule.  This  information  is  important  for  making 
decisions  concerning  allocation  of  capital  resources  and  manpower.  To  use 
CPM,  it  is  necessary  to  know  the  complete  breakdown  of  the  project  tasks 
(or  activities),  their  interdependencies,  and  their  duration  times.  This 
information  is  presented  in  a  network  format.  CPM  then  identifies  the 
sequence  of  activities  that  are  most  critical  to  determining  the  earliest 
project  completion  date.  This  is  done  by  means  of  computing  the  earliest 
data  at  which  an  event  can  occur  without  extending  the  project  completion 
date.  The  difference  between  the  two  times  is  called  slack  time.  Any 
delay  in  any  activity  in  this  sequence  will  also  induce  project  delay  bv 
the  same  amount  PERT  is  similar  to  CPM  in  purpose  and  method  of  analysis. 
The  only  basic  difference  is  that,  in  PERT,  the  duration  time  for  each 
activity  is  a  random  variable  of  known  (or  assumed)  probability 
distribution,  whereas  it  is  treated  as  a  known  deterministic  parameter  in 
CPM.  It  is,  however,  assumed  that  the  most  optimistic  estimate  a,  the  most 
likely  estimate  m,  and  the  most  pessimistic  estimate  b  of  each  activity 
duration  time  are  available.  By  assuming  the  beta  distribution,  the  mean 

te  and  the  variance  Ve  of  an  activity  duration  time  can  be  estimated  as 

9 

a+4m+b  —  /b-a 


and  =  ( 
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From  here,  the  computation  and  analysis  proceed  in  the  same  manner  as  in 
CPM  to  determine  a  critical  path  based  on  the  mean  value  of  each  activity 
duration  time.  In  PERT,  however,  probabilistic  statements  about  the 
project  completion  date  (hence,  the  project  delay)  are  possible  through 
some  additional  simulation  work.  Special  analyses  that  can  be  done  with 
CPM/PERT  are  time-cost  trade-off  analysis,  resource  leveling  and  project 
cost  monitoring  and  control. 

Since  their  inception  around  the  late  1950s  (for  both  CPM  and  PERT), 
they  have  been  applied  quite  extensively.  CPM  was  developed  (1956-1959) 
and  used  jointly  by  the  DuPont  Company  and  Remington  Rand  Univac  (see 
Kelly,  1961),  while  PERT  was  developed  in  1958  for  the  Navy's  Polaris 
missile  system  project.  Both  of  these  methods  have  since  been  extensively 
applied  in  research  and  development  scheduling,  construction  planning,  and 
resource  allocation. 

CPM  and  PERT  are  useful  for  identifying  bottlenecks  and  trouble  spots 
in  a  program.  However,  only  limited  risk  analysis  can  be  done, if  it  can  be 
done  at  all.  Despite  the  inclusion  of  uncertainty  in  time,  PERT  can  at 
best  be  considered  as  a  tool  for  quantifying  risk  with  regard  to  the  time 
element  only.  Moreover,  the  use  of  three  time  estimates  and  the  assumption 
of  a  beta  distribution  have  drawn  considerable  criticism  (Malcolm  et  al., 
1959;  MacCrimmon  and  Ryavec,  1964;  Hartley,  1966). 

2)  Graphical  Evaluation  and  Review  Technique  (GERT)  (Pritsker,  1977; 
Pritsker  and  Happ,  1966;  Pritsker  and  Whitehouse,  1966;  and  Pritsker  and 
Sigal,  1883).  We  note  that  PERT  is  normally  applied  to  projects  with  rigid 
network  structures.  To  apply  PERT,  all  activities  as  well  as  their 
interrelationships  must  be  known,  with  the  understanding  that  the  project 
is  completed  if  all  activities  are  completed.  Moreover,  an  event 
represented  by  a  node  in  the  network  can  occur  if  and  only  if  all 
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activities  leading  to  the  event  are  completed  and  all  activities  following 
this  event  cannot  be  started  until  the  event  occurs.  GERT  was  introduced  in 
1966  to  circumvent  these  and  other  shortcomings  of  PERT.  Indeed,  GERT  was 
developed  as  a  network  planning  and  management  tool  to  deal  with  a  project 
described  by  a  stochastic  (probabilistic)  network  structure  in  which 

*  activities  have  a  probability  of  occurrence,  have  a  choice  of 
being  perormed,  and  have  variable  completion  times 

*  some  intermediate  activities  may  not  be  successful 

*  an  event  may  take  place  provided  that  at  least  one  activity 
leading  to  it  is  completed 

*  repetition  of  an  activity  is  permitted 

*  parallel  or  sequential  activities  may  be  required  in  order  to 
reach  a  goal 

*  alternative  activities  may  be  available  and  specified 

*  a  number  of  possible  outcomes  may  occur 

GERT  is  very  useful  and  flexible  for  treating  the  element  of  risk  and 
uncertainty  associated  with  two  major  project  variab les - - t ime  and  cost. 
Its  conceptual  bases  are  PERT,  stochastic  network  theory,  the  moment¬ 
generating  function,  flowgraph  theory,  and  simulation.  To  apply  GERT  to  a 
project  it  is  necessary  to  construct  a  corresponding  stochastic  network 
characterized  by  logical  nodes,  probabilistic  realization,  and  additive 
stochastic  parameters  on  the  transmittance. 

Each  node  of  a  stochastic  network  consists  of  a  logical  input 
component  and  a  probabilistic  or  deterministic  output  component.  In  order 
for  a  node  to  be  realized,  the  logical  condition  ( exclusive--OR, 
inc lus ive- -OR, AND )  must  be  satisfied  by  all  of  the  branches  leading  into 
the  node.  All  branches  leading  out  of  the  deterministic  node  are  certain 
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to  occur  if  the  node  is  realized.  If  the  node  is  probabilistic,  then  only 
one  branch  is  followed,  with  a  probability,  p,  of  being  followed.  Two 
types  of  parameters  can  be  transmitted  along  a  branch:  1)  the  probability 
of  taking  a  given  path  and  2)  the  distribution  of  the  time  to  traverse  a 

path . 

Once  a  GERT  network  has  been  built,  moment- generat ing  functions  are 
used  to  analyze  the  system.  The  results  of  this  procedure  yield:  (i)  the 
probability  that  a  specific  node  is  realized  and  (ii)  the  moment- genera t ing 
function  of  the  time  associated  with  a  node  if  it  is  realized. 

GERT  has  been  widely  used  and  applied  to  such  projects  as  those 
involving  queing  systems,  inventory  systems  and  marketing  and  reliability 
analysis  as  well  as  project  management.  For  example,  GERT  has  been  applied 
to  planning  a  marketing  research  project  involving  oil  company  sales 
negotiations  and  market  research  (Moore  and  Clayton,  1976).  In  another 
application,  GERT  was  applied  to  help  balance  resources  and  work  load  in  a 
major  jet  engine  overhaul  unit  at  Kelly  AFB  in  Texas. 

GERT  is  very  versatile  and  makes  possible  a  comprehensive  analysis  of 
a  very  complex  fluid  stochastic  network.  Computer  codes  of  modern  versions 
of  GERT  (e.g.,  Q-GERT)  are  also  readily  available.  This  versality  of  GERT 
is  achieved,  however,  at  a  cost  of  greatly  increased  complexity  both  in 
terms  of  its  theoretical  basis  and  the  tasks  to  be  performed  to  apply  the 
method.  GERT  is  a  rather  formal,  rigorous  tool  characterized  by 
generalized  logic  structure. 

3)  The  Synergistic  Contingency  Evaluation  and  Response  Technique  (SCERT) 
(Chapman,  1979). 

SCERT  is  one  of  the  modern  network-based  tools  that  attempts  to 
integrate  active  risk  assessment  and  management  mechanisms  in  large  scale 
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project  management.  It  is  developed  by  drawing  upon  key  characteristics  of 
cash  flow  ,  the  decision  tree,  and  the  semi-Markov  process.  The  resulting 
mathematical  basis  is  a  special  case  of  GERT,  although  the  overall 
methodology  is  quite  different.  According  to  Chapman,  the  basic 
motivations  for  developing  SCERT  are 

*  the  need  to  consider  risk  assessment  and  management  in  a  direct  and 
active  way  at  a  level  of  detail  appropriate  to  all  the  necessary 
decisions,  avoiding  the  passive  and  biased  measurement  approaches  of 
most  PERT  based  techniques 

*  the  need  for  a  comprehensive  view  of  risks,  whether  or  not  they  are 
put  in  probabilistic  terms,  if  risk  measurement  is  to  be  meaningful 

*  the  need  for  a  synergistic  framework  involving  the  efficient  use  of  a 

range  of  special  expertise  with  a  minimum  of  communication 

interpretation  problems 

The  main  idea  of  SCERT  is  to  identify  all  possible  risks  (including  all 
four  project  risk  variables--cost ,  time,  performance,  and  resources)  and  to 
preplan  actions  to  be  taken  should  an  emergency  actually  occur.  The  risks 
and  corresponding  actions  are  based  upon  the  integration  of  experts 
opinions . 

There  are  four  main  phases  in  the  procedure  outlined  in  SCERT:  (1) 
scope,  (2)  structure,  (3)  parameter,  and  (4)  manipulation  and 
interpretation.  The  scope  phase  consists  of  identifying  all  activities, 
the  primary  risks  and  primary  responses,  and  the  secondary  risks  and 
secondary  responses.  These  are  recorded  on  data  sheets.  The  structure 
phase  consists  of  identifying  and  clarifying  relationships  between  risks 
and  ten  ponses  identified  in  the  scope  phase.  This  includes  major  and  minor 
risk  classification  and  general  specific  response  classification.  These  are 
then  diagrammed  in  a  network  similar  to  those  used  in  PERT.  The  analysis 


B-47 


does  not  continue  as  in  PERT,  however;  instead,  a  stochastic  decision 
tree,  which  reduces  to  a  semi-Markov  process,  is  formed.  This  diagramming 
process  stimulates  consistency  and  completeness  checks  and  encourages 
simplifications.  The  parameter  phase  includes  desired  parameter 
identification,  scenario  identification,  and  probability  estimation.  This 
associates  cost  and  time  deviations  from  the  base  plan  and  probabilities 
with  risk/ response  combinations  to  be  modeled  in  a  probabilistic  manner. 
The  manipulation  and  interpretation  phase  consists  of  four  steps:  risk 

computation,  risk  efficiency,  risk  balance,  and  budget  contingency 
assessment.  The  first  three  steps  involve  determining  the  risks  and  the 
relationships  to  expected  costs.  The  last  step  consists  of  determining  an 
appropriate  estimate  for  total  costs. 

SCERT  is  still  in  the  development  stage,  although  the  basic  framework 
and  approach  have  been  laid.  Nevertheless,  it  has  been  applied  to  three 
large  projects:  (i)  an  assessment  of  risks  associated  with  alternative 
construction  schedules  for  a  gas  pipeline  from  the  high  Arctic  to  the 
Canada- US  border,  (ii)  an  assessment  of  the  risks  associated  with 
alternative  bid  packages  for  a  fixed-price  contract  to  construct  a  thermal 
power  station  in  Iraq,  and  (iii)  an  assessment  of  risks  associated  with  the 
North  Sea  pipeline  project. 

Risk  assessment  and  management  provided  in  SCERT  is  mostly  qualitative 
and  it  involves  eliciting  and  aggregating  the  options  of  experts.  A  great 
amount  of  paperwork  and  effort  should  be  anticipated. 

4)  The  Venture  Evaluation  and  Review  Technique  (VERT )  (Moeller  and 
Digman ,  1981 ) . 

VERT  is  a  computerized,  mathematical,  simulation-based  network 


technique  designed  to  systematically  assess 


the  risks  involved  in 
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undertaking  a  program  or  project.  Risk  is  analyzed  with  respect  to  three 
parameters  that  are  of  most  concern  to  managers  of  new  projects:  time, 

cost,  and  performance.  This  makes  VERT  more  powerful  than  other  network 
techniques  that  consider  only  time  and  cost.  It  is  useful  as  an  aid  in 
decision  making  in  situations  with  incomplete  information  about  all 
alternatives . 

The  logical  basis  for  VERT  is  similar  to  that  of  other  standard 
network-based  techniques  (e.g.,  PERT)  except  for  the  following  special 
features:  (i)  six  new  types  of  node  logics  are  introduced  to  allow  greater 

flexibility  and  capability  in  modeling,  (ii)  thirteen  statistical 
distributions  are  provided  to  model  time,  cost,  and  performance  as  random 
variables  and  (iii)  mathematical  relationships  are  introduced  to  relate 
time,  cost,  and  performance  analytically  for  the  purpose  of  simulation. 

Two  symbols  are  used  in  forming  a  VERT  network:  (1)  nodes-- 

character ized  by  a  certain  input/output--representing  milestones  or  decision 
points  and  (2)  lines  representing  activities  that  are  characterized  by 
three  parame ters- - the  time  consumed,  the  cost  incurred,  and  the  performance 
generated  in  completing  this  activity.  The  values  of  these  parameters  can 
be  assigned  in  terms  of  a  standard  statistical  distribution,  a  histogram, 
or  a  mathematical  relationship  depending  on  other  nodes  or  arcs.  Once  the 
network  has  been  constructed,  VERT  simulation  traverses  from  the  initial 
node ( s )  to  the  final  node(s)  to  create  one  trial  solution.  This  process  is 
repeated  a  sufficiently  large  number  of  times  in  order  to  find  a 
probabilistic  indication  of  possible  outcomes  of  the  project,  including 
success  or  failure. 

Applications  of  VERT  include  a  weapons  system  development  project 
(Thomas,  1977;  Moeller,  1979;  Brown,  1975),  flood  control  programs, 
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pollution  abatement  methods,  earthquake  analysis,  rail  yard  switching 
operations,  fault-tree  analysis,  production  line-balancing,  and  war  gaming 
(Moeller,  1979;  Moellr  and  Digman,  1981). 

5)  The  Work  Plan  Analysis  and  Scheduling  Tectmiaue  UoPASTl  (De lgrasso  and 
Rosenbluth,  1976). 

This  technique  was  developed  to  plan  and  schedule  complex  engineering 
programs  when  early  goals  and  ideas  are  still  fluid.  The  objective  of  this 
technique  is  to  combine  enhanced  communications  with  PERT  network  analysis. 

A  WoPAST  network  is  formed  in  a  manner  similar  to  PERT/CPM.  In 
addition,  WoPAST  amplifies  dependencies  and  responsibilities  bv 
identification  of  inputs  and/or  information  necessary  to  perform  tasks 
depicted  in  the  network.  A  summary  of  the  application  of  WoPAST  can  be 

outlined  in  eight  steps: 

a)  Define  program  objectives. 

b)  Define  and  list  responsibility  elements  necessary  to  meet 
objectives . 

c)  Assign  personal  accountability  by  responsibility  element. 

d)  Generate  first-pass  action  and  dependency  plans  by  element. 

e)  Optimize  element  schedules,  reconcile  element  interdependency 
mismatches  and  establish  a  critical  path. 

f)  Skew  elements  with  respect  to  the  critical  path.  This  determines 

the  program  completion  date. 

g)  Adjust,  as  required,  all  element  plans  to  achieve  calendar 
synchronization  for  the  total  job.  The  critical  path  should  be 
the  same  as  in  (f). 

h  Utilizing  the  above  plan  as  a  format,  hold  regular  review 
meetings  to  ascertain  status  and  problems. 

WoPAST  helps  to  assign  responsibility  for  different  tasks  to 
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individuals  or  departments  and  enhances  the  communication  and  coordination 
among  various  working  groups.  It  also  provides  for  detailed  monitoring  of 
the  progress  of  the  program  since  areas  of  responsibilities  are  clearly 
defined.  WoPAST,  however,  does  not  have  an  explicit  treatment  of  risks  or 
uncer  ta int ies . 

WoPAST  has  been  claimed  to  have  been  successfully  applied  in  several 
engineering  development  programs  as  well  as  in  medical  research. 

6)  The  Risk  Information  System  and  Cost  Analysis  ( RISCA)Me  thod 

This  is  a  modified  network  analysis  method  that  quantifies  development 
time  and  cost  risk.  RISCA  takes  advantage  of  network  analysis,  simulation, 
and  Monte  Carlo  techniques  in  order  to  account  for  risk  as  a  function  of 
time.  The  procedure  that  RISCA  uses  can  be  summarized  as  follows. 

1)  The  system  is  represented  as  a  probabilistic  model  of  events. 

2)  Monte  Carlo  simulation  is  used  to  generate  sample  distributions  of 

cost  and  completion  time. 

3)  The  user  must  then  analyze  these  distributions  in  the  context  of 

available  alternatives  in  order  to  quantify  the  risk  involved 
with  each. 

RISCA  provides  a  framework  for  analyzing  complex  systems  that 
do  not  lend  themselves  to  conventional  mathematical  determina¬ 
tion.  Another  advantage  that  RISCA  provides  is  the  ability  to 
account  for  information  that  is  gained  over  the  program  life  cycle 
because  it  deals  with  risk  as  a  function  of  time.  The  disadvantages 
of  the  RISCA  method  are  (1)  it  is  hard  to  deal  with  the  inter¬ 
dependencies  involved  in  the  acquisition  process,  and  (2)  it  does 
not  deal  with  the  risk  involved  with  the  performance  parameter. 
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7)  The  Total  Risk  Assessing  Cost  Estimate  (TRACE_) 

This  is  a  method  of  determining  the  cost  of  uncertainties  in  order  to 
include  it  in  the  budgeting  process.  TRACE  provides  the  PM  with  a 
disciplined  approach  for  costing  the  risk  involved  with  advanced 
technological  impacts  on  a  program.  The  process  involves  taking 
uncertainties  into  account  when  estimating  activity  budgets  and  using  the 
most  likely  situation  estimates  as  the  estimate  for  the  activity.  In 
particular,  TRACE  estimates  the  most  likely  cost  due  to  risks  involved  with 
high  technology.  This  estimate  is  then  added  to  the  baseline  cost  estimate 
to  determine  the  total  estimated  cost  of  an  activity.  The  model  used  to 
determine  the  TRACE  estimate  is  a  simulation  model  of  the  contractors’  work 
breakdown  structure  in  combination  with  some  powerful  network  models  (e.g., 
VERT,  RISCA,  TRACENET,  RISNET,  etc.).  The  procedure  includes 

1)  identification  of  major  subsystems  and  unfunded  technicological 
risk  and  uncertainty  from  the  WBS 

2)  classification  of  risks  into  eight  categories  in  order  to  apply 
conventional  cost  estimating  techniques 

3)  assumption  of  independence  between  risks  and  development  of  cost 
estimates  and  probability  distributions  for  each  contingency 

4)  use  of  VERT  and  Monte  Carlo  techniques  to  derive  a  single  cost 
distribution  for  the  program 

5)  application  of  inflation  factors 

TRACE  has  been  applied  extensively  by  the  Army  on  such  programs  as  the 
Pershing  II,  Stinger/Post,  and  the  Remotely  Piloted  Vechicle  program.  This 
method  helps  to  identify  the  costs  of  uncertainty  and  risk  of  a  program 
before  approval. 

8)  Other  Network  Mode  1 s 


The  risk  management  model  developed  by  Lewis  R.  Ireland  of  SWL,  Inc. 
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is  designed  to  help  identify,  assess,  and  manage  the  risk  involved  with  DoD 
system  acquisition  programs.  The  model  takes  advantage  of  network  analysis 
and  probabiity  estimation  techniques  to  address  elements  of  risk  and  to 
work  in  connection  with  commonly  accepted  project  management  practices. 
The  model  uses  the  contractors'  work  breakdown  structure  to  develop  an 
interdependent  network  of  activities.  A  network  analysis  technique  is  then 
used  to  quantify  the  project  completion  time  and  the  start  and  finish  times 
of  each  activity.  The  decision  maker  must  then  set  performance,  completion 
time,  and  cost  criteria  for  each  activity  along  with  a  risk  threshold  under 
which  he  deems  risk  acceptable.  Results  from  the  network  analysis  are  used 
to  assess  the  probability  and  consequences  of  failure  to  meet  the  criteria 
that  were  set.  The  probability  and  consequences  are  then  compared  to  the 
risk  threshold,  and  trade-off  options  are  evaluated  to  determine  management 
actions.  The  advantage  of  this  model  over  others  is  that  it  provides  a 
disciplined  method  for  identifying,  assessing,  and  managing  risk  in  the 
context  of  project  management.  However,  as  the  number  of  activities 
increase  it  becomes  more  and  more  difficult  to  determine  the  network 
relationships, and  criteria  and  threshold  estimates  become  biased. 

The  Mult iple- Incentive  Contract  Evaluation  (MICE)  method  is  used  by 
defense  contractors  to  evaluate  the  risk  relative  to  the  contract.  MICE 
helps  to  identify  feasible  system  parameter  specifications  and  selects  the 
optimal  contract  parameter.  This  method  uses  the  probability  distribution 
of  performance,  cost,  and  scheduling  to  evaluate  the  contract  with  respect 
to  certain  contract  specifications  in  order  to  determine  profitability. 
MICE  is  a  network  analysis  method  specialized  to  the  needs  of  the 
contractor;  however,  it  can  provide  the  PM  with  expectations  about  the 


performance  of  the  contract. 
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Table  B 3  summarizes  and  contrasts 
the  network-based  methods  discussed  in 


the  main  characteristics 
this  section. 
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Table  B.3.  Network-Based  Methods  and  Their  Characteristics 


appendix  c 
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(i)  RISK  OF  IMMATURE  DESIGN 

In  most  programs,  the  result  of  research  and  development  efforts  is  a 
technical  data  package  (TDP)  that  is  used  in  preparing  for  initial 
production.  Therefore,  the  quality  and  thoroughness  of  the  TDP  have  a 
significant  influence  on  costs.  More  specifically,  the  TDP  affects  the 
first  unit  costs  as  well  as  the  learning  curve  related  to  production.  The 
cost  risk  is  calculated  by  determining  the  area  between  the  planned 
learning  curve  and  the  actual  learning  curve.  (From  Ingalls  1984  a.) 


COST 


This  cost  risk  can  be  reduced  by  dedicating  more  effort  toward  the  TDP,  in 

order  to  provide  a  more  detailed  and  complete  production  plan.  This  can  be 

accomplished  through:  (From  Ingalls  1984  a.) 

use  of  production  engineers  in  design 

.  completion  of  PEP  studies 

completion  of  maturity  phase  studies  on  item 

design 
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.  review  of  design  drawings 

.  completion  of  producibil ity  readiness  reviews 
Funding  of  these  activities  before  production  begins  reduces  the  subsequent 
overall  unit  cost  of  production. 

(ii)  SOFTWARE  RISK 

Software  has  become  a  growing  part  of  most  advanced  systems.  Unlike 
hardware,  the  errors  involved  with  software  are  not  readily  seen  by  the 
user.  They  appear  as  incorrect  logic  commands  and  become  much  more 

difficult  to  detect  and  correct  as  time  goes  on.  The  risks  associated  with 
software  components  include  aspects  of  cost,  scheduling,  and  reliability 

factors  that  make  them  very  important  to  the  overall  program.  The  risk 

handling  techniques  used  for  software  analysis  are  pencil  and  paper 

analyses,  simulation  and  system  verification.  These  activities  can  become 
very  costly.  Therefore,  when  allocating  funds  for  software,  the  following 
practices  become  important:  (From  Ingalls,  1984  a.) 

.  Bring  on  software  maintenance  and  support  personnel  early. 

.  Assure  standardization  of  all  documentation  to  ensure 
supportability . 

.  Have  operational  testers  check  whether  the  software 
successfully  meets  the  operational  requirements. 

.  Examine  software  for  possible  incremental  release 
(modular  design). 

If  emphasis  is  put  on  these  areas  early,  a  PM  can  reduce  the  likelihood  of 
future  software  problems. 

( iii)  RISK  FROM  CHANGES  IN  PRODUCTION  RATES 

A  program  may  be  required  to  undergo  a  reduction  in  production  funds 
for  a  particular  period  of  time,  resulting  in  lowered  production  rate. 
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However , 


However,  this  action  induces  a  cost  risk  in  the  form  of  an  increase  in  the 

unit  cost.  To  account  for  this  effect,  the  unit  cost  is  separated  into  two 

components  —  direct  costs  and  fixed  costs.  Direct  costs  involve  all  costs 
that  can  be  assigned  to  a  particular  unit,  while  fixed  costs  include  all 

facility,  equipment,  and  support  costs  which  transcend  the  costs  directly 

assigned  to  all  units  produced.  Because  the  fixed  costs  are  usually 

predetermined,  when  a  cut  in  production  funds  occurs,  only  the  direct 

costs  are  affected  and  the  production  rate  decreases.  This  results  in  the 
necessity  to  reallocate  fixed  costs  across  fewer  units,  yielding  a  higher 

unit  cost  per  item.  For  example:  (adapted  from  Ingalls,  1984  a) 

Facilities,  equipment  for  rate,  support,  etc.,  per  year  ...$200M 
400  items,  materials,  labor,  and  other  direct  @$600,000...  240M 

$440M 


A  reduction  of  $200M  would  all  come  from  the  direct  cost  initially  and 
would  leave  $40M  for  direct  cost  of  items,  which  would  cover 
(40M/600,000  ea)  or  66  items  (assuming  the  vendors  will  still  sell 
materials  at  the  same  rate) 

The  original  cost  per  item  was  $440M  for  400  items  or  $1.1M  per  item. 
Now  the  cost  is  $240M  ($200M  for  support  and  $40M  direct)  for  66 
items.  The  cost  per  item  now  stands  at  $240M/66  =  $3.64M  which 
compared  to  the  $1.1M,  represents  a  330%  cost  growth. 

Over  the  years,  program  managers  have  been  using  the  learning  curve  to  deal 
with  this  risk.  However,  in  recent  years  the  production  rate  curve,  which 
is  more  a  function  of  fixed  costs,  has  been  acknowledged  as  being  a  better 
estimate  of  this  risk.  Therefore,  knowing  the  impact  of  the  production-rate 
curve  on  a  production  cut  will  result  in  the  best  allocation  of  funds. 
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(iv)  RISK  OF  INADEQUATE  FACILITY  START-UP  FUNDS 

Facility  start-up  costs  include  the  cost  of  tooling  and  test  equipment 
that  will  be  needed  during  the  production  phase  of  a  program.  These  funds 
are  usually  required  in  advance  of  production  because  of  the  lead  time 
associated  with  designing,  acquiring,  and  setting  up  the  equipment.  The 
risks  involved  with  these  activities  usually  show  up  as  deviations  in  the 
production  rate.  Generally,  facilitization  equipment  is  acquired  at  a 
level  to  support  some  predetermined  production  rate.  If  funding  for  these 
activities  is  cut  or  under-emphasized,  the  planned  production  rate  may  not 
be  attainable.  This  will  result  in  schedule  slippages  as  well  as  unit  cost 
overruns . 

(v)  RISK  OF  INADEQUATE  LONG-LEAD  FUNDS 

Long-lead  funds  are  required  for  the  procurement  of  materials  and 
parts  that  have  unusually  long  lead  times.  This  aspect  is  of  particular 
interest  to  the  Air  Force  because  of  the  technologically  advanced  materials 
and  parts  required  by  the  aerospace  industry.  Again,  the  risks  involved 
with  this  activity  affect  the  production  rate.  Insufficient  funding  will 
cause  a  reduction  in  the  production  rate,  delays  in  delivery  of  finished 
units,  or  even  a  total  production  shutdown. 

(vi)  RISKS  ASSOCIATED  WITH  LOGISTICS  SUPPORT  ANALYSIS 

Logistics  support  analysis  (LSA)  is  a  process  that  ties  design 
concepts  of  a  system  to  support  concepts  of  the  system.  Data  resulting 
from  LSA  is  provided  to  the  logistics  command  and  allows  for  the 
identification  and  planning  of  required  equipment,  manpower,  technical 
data,  supplies,  and  facilities  to  be  used  in  support  of  the  system.  The 
risks  involved  with  logistics  support  do  not  show  up  in  the  development  or 
production  phase  of  a  program;  yet,  once  a  system  is  deployed,  it  has  a 
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direct  effect  on  the  life  cycle  cost  (LCC)  of  a  system.  The  relationship 
between  LSA  and  LCC  is  a  result  of  the  design  and  supportability  of  a 
system.  Therefore,  a  high  level  of  preparedness,  resulting  from  detailed 
logistics  support  analysis,  will  in  the  long  run  reduce  the  life  cycle  cost 
of  a  system. 

(vii)  RISKS  ASSOCIATED  WITH  SYSTEMS  AVAILABILITY 

Systems  availability  can  be  thought  of  in  terms  of  two  aspects, 
maintainability  and  usability.  Maintainability  depends  on  the  amount  of 
spare  parts  and  support  equipment  available  to  the  system,  while  usability 
depends  on  the  amount  of  training  equipment  available  along  with  the 
maintainability.  Although  the  relationship  between  availability, 
maintainability,  and  usability  is  complex,  a  certain  minimum  level  of  spare 
parts,  support  equipment,  and  training  equipment  is  required  in  order  to 
achieve  and  sustain  a  particular  level  of  system  availability.  Again,  the 
risk  involved  with  systems  availability  is  not  realized  until  the  systems 
have  been  deployed;  however,  systems  availability  may  have  a  significant 
effect  on  our  national  defense. 
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This  approach  to  risk  quantification  is  based  on  the  concept 
of  conditional  expectation.  The  resulting  set  of  functions 
provides  a  fuller  description  of  risk  than  that  of  the  more 
popular  expected  value.  Employing  these  risk  functions  in 
conjunction  with  the  surrogate  worth  trade-otf  method- 
a  multiobjective  decision-making  methodology  -  allows 
decision-makers  to  rigorously  consider  risk-related  policy 
decisions  without  explicit  expression  of  their  utility  functions 
or  explicit  knowledge  of  relative  objective  function  weights. 
An  example  employing  Monte  Carlo  techniques  develops  the 
method  stepwise  i 

l.  Literature  review 

/./  The  risk  assessment  process 

A  common  trend  in  the  quantification  of  risk 
functions  is  the  use  of  the  expected  value  func¬ 
tion.  This  approach  melds  events  corresponding 
to  all  degrees  of  loss  and  probability  of  occur¬ 
rence,  extreme  events  in  particular  become  con¬ 
cealed  during  this  amalgamation.  Because  there 
are  some  inherent  disadvantages  in  using  the 
expected  value,  an  alternative  approach  is  pro¬ 
posed  here.  This  approach  builds  on  the 
strengths  of  the  expected  value,  but  offers  a 
fuller  description  of  the  distribution  of  risk  and 
provides  for  an  accounting  of  the  political  and 
socioeconomic  aspects  of  risk  assessment. 


North-Holland 

Large  Scale  Systems  6  (1984)  13-38 


Risk,  a  focal  topic  in  decision  analysis  today, 
has  been  given  many  definitions.  The  U.S.  Water 
Resource  Council  [46]  reserves  risk  for  potential 
outcomes  described  by  well-known  probability 
distributions,  and  uncertainty  for  those  not  well- 
known.  Lowrance  [30]  defines  risk  as  the  prob¬ 
ability  and  severity  of  adverse  effects.  Kaplan  and 
Garrick  [27]  define  risk  as  uncertainty  plus  the 
potential  for  loss  or  damage.  The  discussion  that 
follows  in  this  paper  adopts  the  USWRC  di¬ 
chotomy  of  risk  and  uncertainty  and  Kaplan  and 
Garrick's  definition  of  risk. 

The  risk  assessment  process  is  also  debated. 
Lowrance  [30]  considers  the  measurement  of  risk 
to  be  an  objective,  but  points  out  that  probabilistic 
activity  and  judging  risk  acceptability  involve 
active  personal  and  societal  value  judgment.  The 
Committee  on  Public  Engineering  Policy 
(COPEP)  [8]  describes  a  technical  process  where 
experts  study  alternatives  and  present  explicit 
consequences,  followed  by  a  political  process 
involving  wide  discussion  to  resolve  value 
conflicts.  Kaplan  and  Garrick  [27]  ask  what  losses 
can  happen,  how  likely  are  they  to  happen,  and 
what  are  their  consequences. 

Haimes  [16]  describes  a  three-phase  process. 
Data  collection,  retrieval,  and  processing  occur 
through  active  public  participation.  This  is  fol¬ 
lowed  by  modeling  of  risk  and  other  objectives  and 
the  generation  of  Pareto-optimal  policies  [36]  with 
trade-offs,  after  which  sensitivity  analysis  is  done. 
Finally,  the  analyst  interacts  with  the  decision 
maker(s),  who  use  subjective  value  judgment  to 
select  a  preferred  policy  in  the  context  of  the 
overall  decision. 

Gathering  an  adequate  data  base  is  often  a 
substantial  activity  in  risk  assessment,  diverse, 
long-term  risk  data  has  not  yet  been  specifically 
collected.  No  direct  data  may  even  be  available  for 
new  technologies  [44].  Morgan  [32],  [33]  identifies 
five  data  knowledge  levels:  good  statistical  evi¬ 
dence  for  the  process  is  available;  the  subprocesses 


0167-420X/84/S3.00  ©  Elsevier  Science  Publishers  B.V.  (North-Holland) 


A 


►  1 


E.L  Asbeck ,  Y.  Y  Haimes  /  Partitioned  multiobjectwe  risk  method 


14 

have  good  statistical  evidence:  no  good  data  is 
available,  but  a  similar  process  is  well  known:  only 
professional  intuition  and  judgment  are  available: 
and  even  the  experts  have  little  basis  for  judgment. 
Haimes  [14.16,17]  and  Haimes  and  Hall  [24] 
further  consider  data  inadequacy  and  other  mode¬ 
ling  pitfalls. 

Slack.  Wallis,  and  Matalas  [43]  consider  the 
best  assumption  for  the  distribution  of  a  T- year 
flood  by  relating  expected  design  loss  to  an 
assumed  distribution  given  the  actual  underlying 
distribution.  Four  distributions  (normal.  Gumbal. 
log  normal,  and  Weibul)  are  considered  and 
Monte  Carlo  techniques  are  used  to  determine 
sensitivities  of  the  expectation  to  variations  in 
actual  and  assumed  distributions,  skewness, 
sample  size,  and  year-flood  size.  The  results  of 
this  analysis  are  given  for  three  information 
levels.  With  no  information,  the  normal  dis¬ 
tribution  is  the  best  assumption.  Given 
identification  of  the  actual  distribution,  the 
normal  distribution  is  still  the  best  assumption. 
Further  identification  of  ranges  for  skewness  and 
the  relative  scale,  coupled  with  knowledge  of  the 
expected  design  loss,  can  improve  on  the  normal 
assumption. 

The  marginal  benefit  of  one  decision  versus 
another  is  important  information  in  the  decision¬ 
making  process:  trade-offs  help  guide  decision¬ 
makers’  choices.  Useful  analysis  methods  should 
make  hidden  trade-offs  explicit  (Sage  and  White 
[41]).  An  activity's  societal  benefits  must  balance 
the  trade-offs  of  risks  borne  inequitably  in  a 
population  (Rowe  [40]).  Decision-makers  should 
consider  the  global  circumstances  of  a  decision 
(Morgan  [33]). 

Subjective  judgment  guides  the  choice  of  ac¬ 
ceptable  risk  levels.  Values,  central  to  the  ac¬ 
ceptance  process,  are  evolving,  dynamic  entities 
(COPEP  [K]).  Fischhoff,  Lichtenstein.  Slovic. 
Keenev  and  Derby  [9]  warn  that  values  can  be 
dynamic,  situation  dependent,  and  open  to 
manipulation  during  the  decision  appraisal,  not¬ 
ing  that  issues  of  fact  and  issues  of  value  should 
be  kept  scrupulously  distinct  in  the  decision¬ 
making  process  Given  the  interaction  among 
various  personalities  and  between  individual  and 
group  goals,  ‘any  successful  analysis  must  be 
sensitive  to  the  nature  of  the  decision-making 
process*  (Cohon.  ReVelle  and  Palmer  [  ]). 


1.2.  General  frameworks  for  risk  assessment 

COPEP  [8]  ‘believes  that  risk  and  benefit  must 
be  regarded  as  a  continuum,  and  incremental 
changes  across  the  whole  range  must  be  part  of 
the  analysis'.  ‘There  is  no  such  thing  as  zero  risk* 
(Rowe  [39,40]).  that  is.  no  loss  may  reach  a 
magnitude  where  it  becomes  ‘unthinkable*  to 
allow  any  probability  of  occurrence.  Okrent  [34] 
observes  that  society  is  not  risk  free  and  cannot 
be 

House  [26]  considers  risk  in  the  political 
decision-making  process.  Fischhoff  et  al.  [9]  dis¬ 
cuss  the  uncertainties  in  assessing  the  precise 
level  of  very  low  probabilities  and  define  three 
categorical  approaches  to  acceptable  risk:  pro¬ 
fessional  judgment,  bootstrapping,  and  formal 
analysis. 

Professional  judgment  integrates  facts  and 
technical  experts*  values.  Bootstrapping  assumes 
that  evolutionary,  adjustive  processes  have  in  the 
past  struck  a  societallv  acceptable  balance  that 
is  a  useful  guideline  to  future  decisions.  One 
approach  based  on  the  bootstrapping  philosophy 
(Starr  [44])  determines  what  is  ‘traditionally  ac¬ 
ceptable  rather  than  ‘best  Formal  analysis  ap¬ 
plies  analytic  schemes  from  formally  defined 
rational  principles,  the  central  question  is  to 
determine  at  what  point  risk  is  ‘sufficiently* 
reduced. 

1.3.  Formal  analysis 

Formal  analysis  can  help  in  clarifying  the 
questions,  making  underlying  assumptions  ex¬ 
plicit.  anticipating  consequences,  and  describing 
trade-offs  and  options  (Lowrance,  in  Schwing 
and  Albers  [42]). 

1.3.  /  Utility  theory  approaches 

Many  formal  analysis  approaches  are  ap¬ 
plications  of  utility  theory  (see  von  Neumann 
and  Morgenstern  [47]  and  Keeney  and  Raiffa 
[29]).  Friedman  and  Savage  [10]  suggest  a  parti¬ 
cular  utility  curve  to  explain  some  classic  con¬ 
sumer  behavior.  Reutlinger  [38]  notes  that  public 
investment  decisions  require  a  societal  utility- 
function.  but  individual  functions  (being  car¬ 
dinal)  cannot  be  mathematically  aggregated. 
Conversely.  Hax  and  Wiig  [25]  combine  utility 
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functions  by  weighting  factors  in  capital  invest¬ 
ment  decisions,  while  Keefer  and  Pollock  [28] 
aggregate  utility  functions  to  perform  resource 
allocation  with  multiple  objectives. 

Swalm  [45)  investigates  expected  value  by 
suggesting  two  alternatives:  receiving  one  million 
dollars,  or  a  coin  toss  returning  nothing  for 
heads  and  three  million  dollars  for  tails.  Most 
people  will  quickly  choose  the  one  million  dol¬ 
lars,  although  the  expected  value  of  the  com  toss 
is  1.5  million  dollars:  even  a  hve-million-dollar 
chance  is  often  not  preferred,  counter  to  some 
applications  of  decision  theory  where  expected 
value  is  the  decision  criterion.  Many  people 
would  prefer  a  guaranteed  income  of  2x  to  an 
equal  chance  for  an  annual  income  of  v  or  }x 
(Marshall  [31)). 

1.3  2.  Other  approaches  ’ 

There  are  approaches  not  explicitly  based  on 
utility  theory.  Haimes,  Loparo.  Olenik  and 
Nanda  [23]  developed  the  multiobjective  statis¬ 
tical  method  (MSM),  a  statistical  assessment  of 
different  system  configurations  in  a  context  of 
competing  objectives.  Given  a  set  of  decision 
choices  .v,  the  expectation  of  each  objective 
function  f(x)  may  be  determined  as  follows:  for 
each  .v,  an  expected  value  for  f(x)  may  be 
simulated:  a  probability  is  associated  with  each 
result:  summing  products  of  the  expected- 
value/probability  pairs  over  the  range  of  the 
random  variables  for  fixed  v  gives  an  expectation 
for  f(x).  The  surrogate  worth  trade-off  (SWT) 
method  and  its  extensions  (Haimes  and  Hall 
[20]:  Haimes  [15))  are  used  to  develop  trade-offs 
and.  through  interaction  with  the  decision¬ 
maker,  a  preferred  solution  is  obtained. 

Kaplan  and  Garrick  [27]  note  that  multiplying 
probability  times  consequences  to  compute  risk 
equates  low-probability,  high-damage  scenarios 
with  high-probability.  low-damage  ones  - 
not  equivalent  events.  They  view  risk  as  a 
functional  relationship  of  probability  (or.  instead, 
frequency  of  occurrence)  and  consequence 
(damage  or  loss)  rather  than  the  expectation  of 
the  function,  and  present  a  technique  for  finding 
that  functional  relationship.  Howard  (Haimes 
[16])  proposes  the  concept  of  a  micromort  -  one 
one-millionth  chance  of  death -as  a  means  of 
expressing  risk. 


Rarig  and  Haimes  [37]  develop  a  dispersion 
index  and  sensitivity  trade-off  through  a  Tay¬ 
lor's  series  first-order  approximation  and  apply 
them  in  an  algorithm  based  on  the  SWT  method. 
The  uncertainty/sensitivity  index  method  (USIM) 
(Haimes  and  Hall  [21]:  Haimes  [16])  uses  a  sur¬ 
rogate  uncertainty  function  to  describe  the  sen¬ 
sitivity  of  a  dominantly  influential  equation 
parameter  and,  in  the  multiobjective  decision¬ 
making  format  of  the  SWT  method,  develops 
trade-offs  between  a  business-as-usual  option 
and  a  conservative,  risk-averse  option. 

1.4  Perception  of  risk 

Slovic.  Fischhoff  and  Lichtenstein  (Schwingand 
Albers  [42])  and  Fischhof  et  al.  [9]  have  done 
recent  work  in  risk  perception.  They  feel  that 
three  heuristics  are  especially  relevant  to  risk 
perception:  availability,  or  judging  the  likelihood 
of  rare  events  by  the  ease  of  recalling  or  imagin¬ 
ing  them:  overconfidence,  or  underjudging  the 
degree  of  risk  related  to  oneself  and  presuming 
uncertain  quantities  can  be  judged  more  pre¬ 
cisely  than  proves  to  be  the  case:  and  the  desire 
for  certainty,  which  causes  denial  of  uncertainty 
to  reduce  the  anxiety  it  can  generate. 

2.  Mathematical  foundations 

The  partitioned  multiobjective  risk  method 
(PMRM)  (Asbeck  [l])  employs  several  concepts. 
The  theory  of  random  variables  may  be  used  to 
find  an  unknown  marginal  probability  density 
function  (marginal  pdf).  A  conditional  expec¬ 
tation  may  be  defined  using  this  marginal  pdf. 
and  both  may  be  approximated  through  Monte 
Carlo  techniques.  Finally,  the  surrogate  worth 
trade-off  (SWT)  method,  a  multiobjective 
decision-making  technique,  is  valuable  in  risk- 
related  decisions. 

2. 1  Random  variables 

Fapoulis  [35]  describes  a  method  for  finding 
the  unknown  marginal  pdf  of  a  random  variable 
if  the  process  generating  that  random  behavior  is 
known  in  terms  of  random  variables  with  known 
marginal  pdfs.  Consider  the  general  problem  of 
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finding  the  probability  rule  for 


2.2.  Conditional  expectation 


X  =  g(VV  V: . VJ,  (2.1) 

where  the  Y,  are  random  variables  with  known 
probability  rules.  Define  a  set  of  auxiliary  ran¬ 
dom  variables  XV  Xz . Yn  as  follows: 

X.-gdVV  V: . Yn)±g(Yx.  Yz . yj, 

(2.2a) 

x,  =  g.<y,.  y2 . yj*  y,  /  =  2,3 . n. 

(2.2b) 

Furthermore,  define  h(X |.  Y':,  Y\ . y„)  as  the 

solution  of  equation  (2.2a)  for  Yl%  and  then  sub¬ 
stitute  the  identities  X,  =  V',  i  =  2,  3 . n.  The 

joint  pdf  of  the  Y,  is  then: 

f  X\..X:.  ..y„Ui<  xz . *n) 

_  f  y,.  ri.  vv.(yi-  >2 . >’n)  .  (2  2) 

deti(Vl«  V; . \'n  )i  I  V,  =  ,,UI. 

v,  *«,.!»  2.  3.  .  n 

where  J(  •  )  is  the  Jacobian  and  det  J{  • )  *  0.  For 
this  problem: 


det  J(  ■ )  =  dgi/^v, . 


If  it  can  be  assumed  that  the  Y'  are  independent 
random  variables,  then: 


f \j.  Y\  \M  i’  A- . x2) 

=  /v,(Vi )/v ;(>’:)  •  •  •/>„(>’«)  |  ~^j| 


.(2-4) 

I  V|  -  X„l 

v,  ■  *,.  1  ■  2.  3.  .  n 


The  probability  density  function  sought  is  the 
marginal  pdf  f\y(x\).  This  can  be  found  by  in¬ 
tegrating  the  other  n  -  1  variables  xz,  x^ . v„ 

over  their  limits.  In  general’ 


A,(.<  i)  = 


Note  that  if  the  A'„  i  -  2.  3 . n.  are  constrained 

to  intervals  (e.g.  uniformly  distributed),  the  in¬ 
tegral  bounds  become  functions  of  the  remaining 
variables  which  can  create  additional  com¬ 
putational  complexities. 


Conditional  expectation  based  on  a  marginal 
pdf  may  be  defined  as  follows.  Given  the  mar¬ 
ginal  pdf  px(x)  ~  Pr{X  =  x}  governed  by  the 
axioms 

Px(x)2!  0,  -x<*<x 

J  ^Px(x) d.x  =  1  . 

Px(x)=  j  p.v(v)  dy  is  nondecreasing  .  (2.6) 

Pr(a  <  X  <  b}  =  f  p*(A)dA, 

J  a 

and  assuming  p(x)  ^  0  for  0  <  x  <  *  and  p(x )  =  0 
for  the  conditional  expectation  of  an 

event  D  =  (x  x  E  \a ,  b]},  where  the  notation  c  E 
(a.  b ]  means  that  a  <  c  ^  b.  is  given  by 

[  xp,\(x)  dx 

E  ( AT !  O  ]  =  -  (2.7) 

p.x(x)dx 


2.3.  Monte  Carlo  approximation 

Repetitive  computations  can  provide  a  sampling 
of  the  behavior  of  a  random  variable  X 
Given  the  functional  relationship  X  = 

g(5,  yt,  Yz . y„).  where  Yt.  i  ~  1.2...  ,  n. 

are  random  variables  with  known  distributions 
and  5  is  a  variable  related  to  a  policy  option,  a 
model  of  the  process  may  be  constructed.  Fixing 
s ,  successive  calculations  of  X ,  using  randomly 
drawn  values  for  the  Yr  build  a  sample  of  A'  s 
behavior  The  density  distribution  of  this  sample 
can  be  considered  to  be  an  approximation  of 
px(x).  the  probability  density  function  for  X. 
Define  the  following  quantities: 


k  =  total  number  of  simulator  samples, 
x,  =  individual  simulator  sample  values: 
i  =  1.  2 . k  . 

X  =  an  integer;  1  ^  N  <  ^  , 
x  =  max  .v, . 

1  *rsk 

x  =  min  x, . 

I 

A  a  =  (x  -  x  )/N , 


(2.8) 
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(T^a  [))=  the  number  of  x,  such  that  x,  E  (a,  b\ 
or,  if  a  =  x,  .x,  E  [ x ,  b], 
a  <b^x% 

fn  =  cr(x  +■  (n  -  l)A.x,  x  +  nAx) :  l  ^  n  <  N . 

The  probability  density  function  may  be  ap¬ 
proximated  by 


2.4.  The  surrogate  worth  trade-off  (SWT)  method 

The  multiple-objective  optimization  problem 
is  also  known  as  a  vector  optimization  problem 
Two  approaches,  the  parametric  and  the  e -con¬ 
straint  (which  is  employed  in  the  SWT  method), 
are  outlined  here. 


px(x)=>  p(x,)=  -jf* -.  (2.9) 

for  n  such  that  xt  E  (_x  +  (n  -  l)A.x,  x  +  mA.x)  or. 
when  .x,  =  x,  n  =  1.  Since  there  are  a  finite  num¬ 
ber  of  observations,  there  exists  a  8  >  0  such  that 
for  all  Ajc  <  8  and  for  all  1  n  ^  ,V,  fn  E  {0,  1}  thus, 
choosing  lx  <  5  results  in: 

p(.x,)E{0,  1/A: Ax}  .  (2.10) 

The  probability  distribution  (also  called  the 

cumulative  distribution  function -cdf)  may  be 

approximated  by: 

P.*(.x)  =  [  p(v)dy  -  X  p(x,)lx.  (2.11) 

Observing  that  p(x,)lx  E  {0,  ilk)  implies  that, 
for  lx  <  6: 

The  conditional  expectation  of  the  event  D  = 
(jcLx  E  (a,  fc]}  for  x^a<b^x  may  be  ap¬ 
proximated  (see  equation  (2.7))  by: 


2.4.1.  Vector  optimization  problems  and  nonin¬ 
ferior  solutions 

The  vector  optimization  problem  (Bravton  et 
al.  [3|)  defines  a  decision  vector  s  = 

(Sj,  Si . O,  an  objective  vector  /  = 

(/i,/:.  ../*,)  with  /,  :Rn— R1  and  /:.V  — 

and  a  set  of  feasible  solutions  5  =  {s'  g(s )  ^  0} 
with  g  =  (gi,  .  .  ,  gp).  where  Rl  and 

g  \  Rn  —+RP.  The  notation  /:Rn— ♦R'"  means  that 
the  function  f(s)  maps  values  from  the  space  of 
real  numbers  with  dimension  n  into  the  space  of 
real  numbers  with  dimension  m.  Assuming  the  /, 
have  noncoincident  minima,  minimize  them.  A 
point  sE5CRfl  is  a  noninferior  (Pareto)  point 
for  a  mapping  /  if  and  only  if  no  change  As  E  Rn 
exists  such  that,  for  all  /  =  1,2 . m : 

f,{s  +  Is)^f(s) ,  (2.15) 

with  strict  inequality  for  at  least  one  function 
when  s  +  As  E  S  CRn. 

Consider  (Fig.  1)  a  scalar  decision  variable  s 
and  two  conflicting  quadratic  objective  functions 
fx  and  fz  (Haimes  [14])  in  the  decision  space, 
where  the  region  .V  represents  the  noninferior 
solutions.  Those  noninferior  solutions  are  shown 
in  the  functional  (or  objective)  space  in  Fig.  2. 


2  X,p(x,)lx 

E[X|D|  =  2£i^ - 

2  pMSx 


(2.13) 


Observing  that  x,p(x,)lx  E  {0,  xjk)  implies  that, 
for  lx  <  6: 


E[X|  D] 


a  <  x,  s.  b 


cr(a ,  b) 


(2.14) 


Note  that  in  (2,13)  and  (2.14)  the  summation 
bounds  for  a  =  x  are  x  ^  x,  <  b. 


N 

Fig.  1  Two  conflicting  quadratic  objectives  graphed  in  the 
decision  space 
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Fig.  2.  The  noninferior  solutions  from  Fig  I  graphed  in  the 
functional  space. 

To  decrease  the  value  of  one  objective,  the  value 
of  (at  least)  one  other  objective  must  increase. 
This  is  the  essential  quality  of  noninferiority. 

2.4.2.  Parametric  approach 
The  parametric  approach  further  defines  the 
vector  optimization  problem  as 


min  wTf 

i  es 

subject  to  wJe  =  1  . 

w>  38  (=1.2,  . .  m  .  (2  16) 

where  w  =  (w,.  w; . w„)  and  e  =  (1. .  ,  l)G 

This  is  a  well-defined  minimization  problem. 
The  solution  is  unique  when  all  /,  are  convex; 
otherwise,  some  noninferior  solutions  are  un¬ 
obtainable.  For  example.  Fig.  3  shows  a  two- 
objective  minimization  problem  with  a  compact, 
nonconvex.  feasible  decision  set  in  the  functional 
space.  The  noninferior  solutions  are  in  bold 
lines.  Solutions  shown  in  the  heaviest  bold  line  (a 
pocket  of  nonconvexity)  are  not  obtainable  by 
the  parametric  approach,  which  finds  the  mini¬ 
mal-valued  hyperplane  tangent  to  the  convex 
hull  of  the  feasible  set  when  given  a  w  =  (w,.  w^). 
The  hyperplanes  H,  and  H:  in  Fig.  3  both  have 
slopes  corresponding  to  the  same  weights  w  and 
contain  noninferior  solutions;  however,  Ff,  is  the 


Pareto  optimal  solutions  not 
obtainable  by  parametric  approach 


feasible  region  for 
e -constraint  problem 


Pareto  optimal  surface 


feasible  region  with 
g^s)  <  0,  i  =  1,2 . p 


hyperplanes 


Fig.  3.  The  e-constraint  approach. 
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minimal  hyperplane  for  this  h\  so  H2  and  its 
associated  noninferior  solutions  are  never 
obtained  by  the  parametric  approach. 

2.4.3.  The  e-constraint  approach ,  basis  of  the 
SWT  method 

The  e-constraint  approach  clarifies  the  vector 
optimization  problem  differently  Choose  an  i  G 
(12,  and  define  the  problem  (Haimes 

[12-U]): 


min  /,(s) 

subject  to/y (5)  ^  j*  h 


(2.17) 
j  =  1,2 . m  , 


where  each  component  of  /  is  once  continuously 
differentiable  on  S.  From  the  Lagrangian, 


L(s,A)  =  /((5)+2a*/(//(5)-£/)' 
/*< 


pher  A12  related  to  the  Lagrangian  L(s)  - 
f{(s)  +  A,:(/:(s)  ~  £:)•  Chankong  [4]  and  Chankong 
and  Haimes  [5]  give  conditions  relating  the 
parametric  and  £ -constraint  problems.  Haimes  et 
al.  [22]  developed  several  algorithms  for  applving 
the  ^-constraint  approach.  The  PMRM  employs 
the  static  ^-objective  £-constraint  (SNE)  al¬ 
gorithm. 

2.4  4.  The  surrogate  worth  function 

The  surrogate  worth  function  is  defined  as 

W(l  G  [-  10,  +  10],  j  and  i.  j  -  1,2 . m.  For  a 

given  A,/,  there  are  f,(s)  and  f,{s),  i  *  j  and  L  }  = 

1,2 . m,  associated  with  a  particular  sG 

SCR".  The  decision-makers  choose  W„(>.  =  , 
<  )0  when  they  prefer  A,,  units  of  f,(s)  (more, 
equally,  less)  than  1  unit  of  f,(s).  Using  the  W„  and 
the  values  of  f,(s)  and  A„(s),  the  analyst  helps  the 
decision-maker(s)  search  the  noninferior  surface 
defined  by  >  0  until  all  W„  =  0.  Associated  with 
these  Wlf  =  0  are  a  set  of  Aj  and  f*.  Solving  the 
problem 


and  for  each  sER"  with  A„>0  satisfying  the 
Kuhn-Tucker  conditions,  it  follows  that  L(s)  - 
/,(s)  and  e,  =  /,(s);  thus: 

(2  18) 

de,  df. 


min  /,(s) 

sSS 

subject  to /,(*)«/;  .  j*  i  and /  =  1.2 . in. 

(2  20) 

yields  the  preferred  decision  s’ESCR". 


The  £ -constraint  approach  varies  the  e, 
parametrically  to  generate  all  needed  noninferior 
solutions  as  well  as  their  associated  trade-off 
values.  A„.  Haimes.  Hall  and  Freedman  [22]  and 
Chankong  and  Haimes  [5]  discuss  the  SWT 
method  and  the  £-constraint  approach,  while 
Haimes  and  Chankong  [19]  discuss  the  theoretical 
basis  for  the  trade-offs  in  terms  of  the  local  theory. 

Recalling  Fig.  3,  £2  defines  an  artificial  upper 
bound  on  the  feasible  set.  The  associated  s- 
constraint  problem  is: 

min  /i(s) 

i  €S 

subject  to  fz(s)  ^  £2  <  (2.19) 

and  its  solution  is  the  noninferior  point  s*  The 
associated  strictly  positive  trade-off  at  s *  be¬ 
tween  /,  and  fi  is  given  by  the  Lagrange  multi- 


2.4.5.  Strengths  of  the  SWT  method 

The  vector  optimization  problem  may  be 
solved  parametrically  if  the  weights  w,  are 
known,  however,  they  usually  are  not  known. 
The  SWT  method  allows  the  decision-maker  in¬ 
directly  to  discover  the  preferred  weights  by 
searching  the  noninferior  surface  for  a  preferred 
solution.  The  responsibilities  in  the  risk  assess¬ 
ment  process  are  thus  distributed  more  equit¬ 
ably.  The  analyst  obtains,  structures,  and 
presents  the  data.  The  decision-makers  deter¬ 
mine  the  importance  of  the  various  decision  fac¬ 
tors  in  view  of  objective  function  values  and 
trade-offs  expressed  in  familiar  measures. 

Risk-related  decisions  are  often  made  by 
groups.  The  surrogate  worth  trade-off  method 
with  multiple  decision-makers  (Hall  and  Haimes 
[24])  allows  for  using  compromise,  negotiation, 
and  any  quantifiable  decision  rule  in  the  decision 
process. 
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3.  The  method 

The  PMRM  involves  a  six-step  procedure: 

(1)  find  marginal  probability  density  functions; 

(2)  partition  the  probability  axis  to  provide  a 
fuller  risk  description; 

(3)  map  the  probability  partitions  onto  the 
damage  axis; 

(4)  find  conditional  expectations; 

(5)  generate  functional  relationships  between 
conditional  expectations  and  policy  choices;  and 

(6)  employ  the  SWT  method  to  generate 
Pareto  optimal  solutions  and  their  associated 
trade-offs  and  to  choose  a  preferred  policy. 

An  overview  of  the  process  is  presented  in  Fig 

4.  while  Fig.  5  describes  more  detailed  branch 
points  in  flowchart  fashion. 

3. 1  Find  marginal  probability  density  functions 

The  PMRM  requires  the  marginal  probability 
density  functions  (pdfs).  px(x:st).  relating  prob¬ 
ability  of  loss  to  magnitude  of  loss  for  each  of  the 
policy  options  t  -  1,2 . q  The  5,  are  con¬ 

sidered  scalar  in  this  discussion,  although  exten¬ 
sion  to  the  vector  case  should  not  present 
significant  theoretical  difficulties.  These  prob¬ 
ability  density  functions  may  be  explicitly  known, 
obtained  through  random  variable  techniques 
such  as  those  in  Section  2  1.  or  approximated  by 
Monte  Carlo  techniques,  as  in  Section  2.3.  The 
random  variable  technique,  useful  and  inexpen¬ 
sive  in  simple  problems,  is  exact  but  com¬ 
putationally  cumbersome:  the  Monte  Carlo  ap¬ 
proach  is  approximate  but  more  broadly  applic¬ 
able. 

From  these  px(x\s, ),  a  set  of  probability  dis¬ 
tribution  functions  (cdfs)  may  be  defined  as: 

Px(x:  st)=  f  px(y :  s,)dv.  i  =  1.2 . </, 

Jo 

(3.1) 

where  px(x\  s,)-  0  for  x  ^  0.  Each  of  these  cdfs 
is  a  description  of  the  distribution  of  ‘risk’ 
(Kaplan  and  Garrick  [27])  for  the  policy  choice 
s,\  that  is.  the  cdfs  relate  the  loss  .t  and  its 
probability  of  occurrence  px(x:s,)  (Kaplan  and 
Garrick  define  a  relationship  of  loss  and 
frequency  of  loss,  rather  than  probability).  One 


way  to  extract  essential  information  is  through 
mathematical  expectation: 

E[X]  =  |  xpx(x-s,)dx.  (3.2) 

This  condensation  loses  information  about  losses 
at  the  extreme  tails  of  the  loss  distribution. 

3.2.  Partition  the  probability  axis 

The  PMRM  partitions  the  probability  axis  into 
a  set  of  ranges.  The  ultimate  intention  of  this 
partitioning  is  to  provide  the  decision-maker 
with  a  more  complete  view  of  ‘the  distribution  of 
risk.  One  application  concerns  events  that 
represent  extremely  large  losses  with  a  low 
probability  of  occurrence,  while  another  is  con¬ 
cerned  with  describing  optimistic,  middle-of-the- 
road,  and  pessimistic  viewpoints.  Some  guide¬ 
lines  based  on  the  standard  normal  distribution 
N(().  1)  for  choosing  the  partitioning  values  a„ 

'  =  1-2 . n+  1.  on  the  probability  axis  are 

presented  using  Fig  6.  In  the  general  literature 
(e.g.  Section  1).  catastrophic  events  have  10-5  or 
less  probability  of  occurrence;  this  relates  to 
events  exceeding  Ho-  on  N(0,  1).  Employing  the 
N(0.  1)  exceedence  probability  function  1- 
P.x(x  :s,)  as  a  heuristic  (Fig.  7),  it  can  be  seen,  for 
example,  that  if  three  ranges  were  needed  to 
represent  the  bulk  of  the  low-damage  events,  an 
intermediate  damage  range,  and  a  range 
representing  ‘catastrophic'  low  probability 
events,  the  -Her  and  +4 a  partitioning  values 
would  provide  an  effective  rule-of-thumb  in  the 
normal  distribution  case,  the  low  range  contains 
84%  of  the  loss  events,  the  intermediate  range 
contains  just  under  16%  of  the  loss  events,  and 
the  higher  range  contains  about  0.0032%  (or 
3.2 x  10~s  probability)  of  the  loss  events.  Alter¬ 
natively,  using  +2cr  and  Her  as  the  partitioning 
values  results  in  97.7%.  -2.3%.  and  0.0032%% 
for  the  respective  ranges. 

As  another  example,  again  using  the  heuristic 
of  Fig.  7.  the  probability  axis  could  be  par¬ 
titioned  into  optimistic/middle-of-the-road/pes- 
simistic  ranges.  This  could  be  done  by  choosing 
the  partitioning  values  associated  with  ±lcr  for 
the  sample.  This  results  in  the  lower  15.9%  of 
the  damage  observations,  the  middle  68.2%.  and 
the  higher  15.9%. 
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m pdf  marginal  probability  density  function 
jpdf  -  joint  probability  demity  fimclion 
pdf  -  probability  distribution  function 

numbers  in  parentheses  indicate  section*  pertinent  to  that  step 
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Fig  5  Detailed  flowchart  of  the  PMRM  procedure. 
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Fig.  6.  Percent  of  observations  falling  above  and  below  four  cr-1imits 


3.3  Map  partitions  to  the  damage  axis 

Having  decided  the  partition  values  on  the 
probability  axis,  these  values  are  mapped  onto 
the  damage  axis  (as  in  Fig.  7).  Solutions  must  be 
found  to  the  following  problem: 

For  each  partition  value  a„  /  =  I,  2 . n  +  |, 

and  each  policy  option  s,.  j  =  1,2 . q.  find  an 

a„  >0  such  that  P(at,\  s,)  =  a(. 

These  a,,  are  used  in  defining  conditional  expec¬ 
tations  for  the  next  step  of  the  PMRM.  If 
P(x:s,)  has  a  closed-form  expression  for  the 
inverse  (that  is.  there  exists  an  equation  of  the 
form  P~](at:  s,)  =  atj  for  all  a,  and  a*,),  the 
unknown  a„  may  be  found  explicitly;  otherwise, 
those  at,  may  be  found  by  approximation 
through  bisection,  false  position,  or  other  line 
search  techniques. 


3.4.  Find  conditional  expectations 
Conditional  expectations  must  be  found  for 


each 

P{x:  s,).  j  =  I 

— . 

with 

domains  on 

the 

damage  axis 

defined 

by 

the  a„ ,  i  = 

1,2, 

=  1,2, .  .  . 

,  q.  Let 

o„  = 

=  «2/l  . 

/=  1.2... 

.  . q . 

D„  = 

( a„ ,  , 

/  =  2,  3, .  . 

(3.3) 

/=  1.2... 

.q. 

The  expectations  are  computed  (equation  (2.7)) 
to  be: 

a.+i., 

xpx(x\  Sj )  dx 

i  

px(x\  s, )  dx 

- . n  ;  y  =  1.2 . q.  (3.4) 


i 
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Note  that  the  denominator  above  is  actually 
I*  4  px(x\ s,)d.r  -  a^i  -  at ,  /  =  1,  2 . n 


but  the  use  of  the  integral  denominator  reduces 
the  computational  error  arising  from  the  use  of 
approximate  values  for  the  a,  and  a,  in  equation 
(3.4). 

3.5.  Generate  functional  relationships 

Given  the  E[X|  D„],  a  set  of  risk  functions 

f(s)%  i  =  1,2 . n.  may  be  found  as  follows.  If 

it  can  be  assumed  that  the  conditional  expec¬ 
tations  for  values  of  s  between  the  known  data 
points  act  in  a  continuous  and  a  simple  way,  then 
for  any  region  on  the  probability  axis  D(/,  with 

iE  {1.2 . N}.  regression  may  be  used  to  fit  a 

smooth  curve  f,(s)  to  the  point  pairs 

{s„  E[X|DV]},  /=1,2 . q  If  the  continuity 

assumptions  cannot  be  made  or  a  smooth  curve 
cannot  be  found  to  fit  the  data  points  to  the 
analyst's  satisfaction,  the  data-point  pairs  may  be 
used  in  lieu  of  the  /((s)'s  to  obtain  a  less  general 
result  in  the  next  step.  Each  f,(s)  relates  the 
damage  domains  associated  with  the  partitioned 
regions  on  the  probability  axis  to  the  policy 
variable  s. 

3.6.  Employ  the  SWT  method 

In  Section  3.5  a  set  of  risk  objectives  was 
created  that,  in  combination,  can  provide  some 
insight  into  how  risk  is  distributed  over  the  range 
of  losses  for  each  decision  choice.  A  structured 
technique  is  required  for  effectively  employing 
this  information  and  valuing  each  decision 
choice.  Trade-off  information  for  the  decision¬ 
makers)  is  required;  furthermore,  risk  is  only 
one  component  of  the  broader  context  of  the 
decision-making  process.  These  criteria  suggest 
the  necessity  of  a  multiple  objective  decision¬ 
making  methodology  that  allows  decision-makers 
to  express  their  implicit  values  and/or  those  of 
their  constituents  during  the  decision-making 
process;  the  surrogate  worth  trade-off  (SWT) 
method  (Section  2.4)  satisfies  these  needs  by 
providing  trade-offs  among  the  several  objectives. 


Through  the  SWT  method,  the  f(s ),  *  = 

1,2 . n,  may  be  used  in  conjunction  with  a  set 

of  conflicting  nonrisk  objective  functions  /;(5), 
j-  1,  2, ...  m.  and  a  feasible  decision  set  S  = 
{s|g,(s)=sO,  f  =  l,2 . p).  as  follows.  Arbi¬ 

trarily  choosing  the  first  nonrisk  objective  as  the 
primary  objective  (although  experience  has 
shown  that  the  objective  measured  in  monetary 
units  is  the  best  selection  as  the  primary  objec¬ 
tive)  and  for  any  one  risk  function  fH(s)  with 

hE{\.2 . n}%  solve  the  problem  Ph  (see 

below)  to  obtain  trade-offs  between  the  risk 
function  fh(s)  and  the  nonrisk  objectives; 

Ph  .  min  /i(s) 

jES 

subject  to/;(s)  ^  ,  /  =  2,3 . m 

fh(s)^eH .  he{\.  2 - n} . 

In  practice,  the  trade-offs  between  the  m 
conflicting  nonrisk  objectives  need  be  obtained 
onlv  once,  while  the  trade-offs  related  to  each  of 
the  n  risk  objectives  can  be  obtained  by  swap¬ 
ping  one  risk  objective  for  another  in  Ph.  This 
process  of  swapping  the  risk  objectives  one  at  a 
time  is  necessary  because  these  risk  functions  are 
dependent  upon  each  other  by  construction  (see 
Section  4.4).  The  trade-offs  provide  extremely 
useful  information  in  the  decision-making 
process. 

If  the  continuity  assumption  in  Section  3.5 
cannot  be  justified,  the  trade-offs  in  the  SWT 
method  may  be  obtained  by  approximation  of 
the  partial  derivative;  that  is, 

df\(s,) _ /i(J,)-/i(sQ 

A‘*  "  dfh(s,)  E [X\Dh,  s,]  -  E[Xj Dhk.  sk] ' 

h  =  1, 2,  ....  n  ,  (3.7) 

where  k  =  /+  1  for  an  increase  of  s,  to  5/+1  and 
k  =/  -  1  for  a  decrease  to  sf. i.  Gemperline  [11] 
employed  this  approach  with  the  SWT  method. 
Although  heuristicallv  appealing,  technical 
details  of  this  approximation  have  not  been 
confirmed, 

These  trade-offs  allow  decision-makers  to  see 
the  marginal  cost  of  a  small  change  in  an  objec- 
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tive,  given  a  particular  level  of  risk  assurance  for 
each  of  the  partitioned  risk  regions.  A  know¬ 
ledge  of  marginal  costs  gives  the  decision-maker 
insights  that  are  useful  for  determining  accept¬ 
able  risk  levels.  In  general,  trade-offs  between 
the  risk  functions  associated  with  any  one  loss 
dimension  cannot  be  found;  however,  if  more 
than  one  risk  axis  is  used -say  mortality,  mor¬ 
bidity,  dollars  lost,  etc.  -  trade-offs  between  these 
risks  should  be  obtainable  if  the  objectives  are  in 
conflict. 


4.  Comments  and  observations 

4  l.  On  creating  the  risk  functions 

In  the  spirit  of  regarding  risk  as  a  distribution 
of  probability  and  damage,  the  decision-maker 
should  ideally  be  presented  with  the  entire  dis¬ 
tribution  of  risk  for  each  policy  option.  This 
approach  quickly  becomes  confusing  and  cannot 
provide  the  marginal  worth  of  one  decision  over 
another,  nor  can  it  show  the  relations  between 
various  nonrisk  objectives  and  the  risk  aspects  of 
a  decision.  The  PMRM  includes  risk  distribution 
information  through  the  functions  ft(s ),  /  = 
1,2 . n,  that  relate  the  conditional  expec¬ 

tations  associated  with  the  probability  axis  par¬ 
titions  to  the  policy  variable  s  (Section  3.5);  this 
provides  information  across  the  entire  domain  of 
the  damage  t. 

4.2.  Sufficient  Monte  Carlo  samples 

The  Monte  Carlo  sample  size  is  dictated  by  the 
'extremeness'  of  the  events  to  be  considered.  In 
general,  if  events  associated  with  Px(x)  >  t  are  of 
interest  and  a  sample  of  r  points  exceeding  t  are 
necessary  (for  an  extreme  event  domain  estimate 
accurate  to  within  some  confidence  bounds), 
then  r/(l-f)  points  must  be  collected.  For  Q 
policy  choices,  Qr/(l  -  t)  data  points  must  be 
computed.  For  example,  if  events  with  Px(x)^ 
t  =  0.999  are  of  interest,  r  =  10  sample  points  are 
desired  for  estimation  accuracy,  and  0  =  5  policy 
options  are  to  be  considered,  then  5  x 
10/(1  -  0  999)  =  5  x  It)4  sample  points  must  be 
computed.  These  five  samples  (each  containing 


IO4  sample  points)  can  each  be  used  as  an  ap¬ 
proximation  to  the  probability  density  function 

4.3.  Relating  conditional  and  unconditional 
expectations 

A  relation  between  the  conditional  (equation 
(2.7))  and  unconditional  (equation  (3.2))  expec¬ 
tations  may  be  found.  Define  the  following  func¬ 
tions: 

/i(s)=a  nonrisk  function  which  serves  as  the 
primary  objective  function  in  the  f-con- 
straint  format 

/,($)=  the  N  -  l  conditional  expectation  risk 

functions,  i  =  2,  3,  4 . N 

fs  +  j  =  the  unconditional  expectation  risk  func¬ 
tion;  that  is,  the  expected-value  function. 

(41) 

Furthermore.  let  0  =  P(. 

P(jtv+t) =  l  be  partition  values  used  to 
define  the  V  -  1  conditional  expectation  risk 
functions.  Note  that  P(.t()  =  a,  for  the  a ,  in  Sec¬ 
tions  3.2  and  3.3.  with  /  =  2,  3, .  ,  n  •*-  l  and  that 

1  because  the  risk  functions  in  this  example 
begin  with  /:( • ). 

Assuming  that  P{x)  is  a  monotonically  in¬ 
creasing  (vs.  nondecreasing)  function  of  .v,  it  may 
be  observed  that  0  =  ,v:  <  xy  <  •  •  -  <  rv  <  a\.1  = 

+x. 

Define  the  following  constant  weights: 

0,(s)=J  p(x.  s )  d.v  ,  /  =  2.  3 . N.  (4.2) 

Noting  that  #,(*)  is  constant  with  changing  s. 
then 

V 

/v..(s)=V(?/,(s).  (4.3) 

Further  effort  should  be  made  to  find  some 
similarly  simple  relation  between  the  conditional 
and  unconditional  trade-offs,  such  that  A^u  = 
4f[ A21, ..  ,  ANl]. 

4  4.  Castastrophic  losses  and  decision-making 
Using  the  notation  from  Section  4.3,  consider 
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Fie.  8  /s .  the  unconditional  expectation,  and  /s.  the  catastrophic  conditional  expectation  vs.  the  cost  objective,  in  the  functional 
space. 


/v+l(s)  (the  unconditional  expectation),  /v(s)  (the 
conditional  expectation  of  the  catastrophic 
damage  events),  and  /,(s)  (the  cost  function).  Fig. 
8  plots  /'i(.s)  vs.  fs (s )  and  fs.x(s).  Note  that  /v.,(.s) 
characteristically  takes  values  less  than  /\(s). 
When  decision-makers  are  presented  with  a  value 
for  /v (v)  as  well  as/v.  i(.v).  they  are  being  reminded 
that  besides  the  lesser  value  for  fs-\(s)  there  is  a 
nonzero  probability  of  a  major  loss  of  /%(.*): 
therefore,  catastrophic  events  are  considered  as  a 
component  of  the  decision  process. 

For  example,  policy  alternative  s  =  A  gives  the 
resulting  values  of  f\(Af  fs(A).  and  /iV^i(A).  If 
the  business-as-usual  approach  is  followed, 
fs~\(A)  alone  would  be  available  as  the  risk- 
representing  function.  The  nonzero  probability  of 
the  significantly  larger  loss  fs(A )  would  have  been 
ignored  from  the  decision-maker's  point  of  view, 
thus,  this  valuable  information  would  have  been 
lost. 


5.  An  example 

The  example  that  follows  is  based  on  Haimes, 


Asbeck  and  Gupta  [18],  a  study  of  risk  analysis 
for  transport  air  pollutants  done  for  the  Office  of 
Technology  Assessment  (OTA),  U.S.  Congress. 

5./.  Problem  definition 

There  has  been  much  concern  of  late  about 
the  effects  of  acid  ram  on  natural  resources.  The 
suspected  mechanism  involves  pollutants  being 
emitted  into  the  atmosphere  (for  example  by 
industrial  smokestacks),  mixing  and  interreacting 
in  the  atmosphere  while  being  carried  far  from 
the  emission  sources,  raining  down  on  nonin- 
dustrial  wild  areas,  and  subsequently  causing 
ecological  damage  such  as  killing  fish  and  stunt¬ 
ing  tree  growth. 

A  causal  relationship  between  emission  level 
and  resource  damage  was  hypothesized:1 

r  =  C[(l  -  B)EA  +  B  -  T]D/[1  -  T]  ,  (5.1) 

where 

Parameters 

B  =  background  loadings 
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^  =  atm.  chemistry  curve  shape 
T  =  threshold  loadings 
C-  current  resource  damage 
p  =s  dose/response  curve  shape 

Variables 

r  =  resource  damage 
£  =  pollutant  emission  level. 


Resource  damage  is  restricted  to  r  ^  0.  Various 
levels  of  uncertainty  are  associated  with  the 
above  five  parameters,  but  they  are  assumed  to 
inhabit  known  ranges.  Based  on  Slack  et  al  [43] 
all  five  parameters  are  assumed  to  be  normally 
distributed,  independent  random  variables  with 
the  statistics  as  shown  in  Table  l.  Each  mean  is 
the  median  of  the  parameter  s  range,  which  is 
assumed  to  encompass  two  standard  deviations. 
Thus,  the  unknown  random  variable  is  the 
resource  damage  given  as  a  function  of  a  known 
emission  level  and  probabilistic  parameters.  Pol¬ 
lutant  emission  levels  of  0  5,  0.6,  0.7,  0.8,  0  0,  and 
1.0  are  considered.  The  emission  level  E  =  1.0 
represents  current  emissions  and  E  =  0.5 
represents  50%  of  current  emission  levels.  The 
PMRM  is  next  developed  and  demonstrated  step 
by  step. 

5.2.  Find  marginal  probability  density  functions 

5.2.1.  Random  variable  technique 
Define 


x^gdA'B.C.D, 


Table  1 


Parameter 

(random 

variable) 

Mean 

(A) 

Std. 

deviation 

(cr) 

Coeff.  of 

variation 

(ct/m) 

A 

0.8 

0  15 

0.187 

B 

0.1 

0.05 

0.5 

C 

1.0 

0.5 

0.5 

D 

1.0 

0.25 

0.25 

T 

0.25 

0.125 

0.5 

'Dr.  Robert  Fnedman,  Office  of  Technology  Assessment, 
U  S.  Congress. 
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*2  =  g2(  A  B.  C.  D.  E.  T)  =  A  . 
x}  =  g}(A.B.C.D,E.T)=B.  (5.2) 

x4  =  gi(A.  B,  C.  D.  E.  T)  =  D . 
x5  =  gs(A.  B,  C.  D.  E.  T)  =  T . 


The  Jacobian  factor  for  this  problem  (where  C 
acts  as  yt)  is: 


det  J(-)\- 


1 

dr 

dC 

dC 

1(1-  B)EA  +  B-  7T 

[1-7-1 


*0  . 


(5.3) 


Solving  for  C  in  terms  of  the  xt  gives: 


C  =  x,(l  -  *<)[(!  -  x3 )£‘:+  Xj  -  xjI—  =  h(x.  E) . 

(5.4) 

The  marginal  probability  density  functions  of  the 
five  parameters  are  of  the  form. 


/y(  y)  = 


1 

V  2  77  a 


exp 


I 


(5.5) 


Since  the  five  random  variables  are  assumed  to 
be  independent,  the  joint  density  of 
X,,  X2,  X3,  X4,  and  X5  is: 


/vl.X;..V,..Y,.X,(*l.  x 3.  X4.  X,.  E) 

=  x,)[(l-  *,)£*» +  *,-  X5I-I . 

7T~  x  77 

(5.6) 


where 

K(x,  E)  =  t(x2  -0.8)2  +  lO(xj-O.l)2 


-t-  t Xi(l  —  JCs)[(l  —  -t-  Xj  —  JC5]_X*  -  l]2 

+  2(x4  -  l)2  +  4(x5  -  0.25)2 .  (5.7) 


The  marginal  density  function  for  X\  is: 


* 
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E)-  J_x  J  xJ_x  J  x  fxi.X2.XyX4.Xs 
x(xi.  *2*  jc3,  x4.  x<.  E)  dx:d.v3dx4dx5 .  (5.8) 

This  quantity  is  not  explicitly  integrable 
because  of  the  form  of  K{x ,  E).  Given  the 
proper  computer  algorithms  for  approximating 
multiple  integrals,  this  result  could  be  employed 
effectively  in  the  PMRM  mathematics  to  obtain 
numerical  approximations  of  the  expectation  for 
specific  values  of  E.  No  such  computational  tools 
were  readily  available,  so  this  approach  was 
abandoned. 

5.2.2.  Monte  Carlo  approximation 

In  this  approach,  9000  calculations  of  resource 
damage  were  made  for  each  of  the  six  emission 


levels  E  =  0.5  through  E  =  1.0  (see  Section  2.3). 
The  random  parameters  were  assumed  to  be 
independent  and  computations  taking  values  less 
than  zero  were  assigned  the  value  zero.  The 
Box-Muller  transform  (Box  and  Muller  [2])  was 
used  to  generate  normally  distributed  deviates. 
The  samples  were  then  ordered  from  least  to 
greatest.  Fig.  9  shows  histograms  for  the  two 
extreme  cases  (JE  =  0.5,  E=  1.0)  created  using 
resource  damage  increments  of  0.02  and  nor¬ 
malizing  the  occurrences  by  the  sample  size  (9000). 
About  2.5%  of  the  observations  were  zeros. 

5.3.  Partitioning  the  probability  axis 

For  the  purposes  of  example,  both  the  extreme 
event  and  pessimistic/optimistic  approaches  to 
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partitioning  the  probability  axis  (Section  3.2)  are 
employed.  Owing  to  the  limited  number  of 
sample  points,  the  extreme  event  partition  will 
use  0.9  and  0.995,  which  relate  closely  to  the 
normal  distribution  heuristic's  values  of  +lcr  and 
+  3cr.  The  pessimistic/optimistic  values  were 
0.159  and  0.841  (see  Section  3.2). 

5.4.  Map  partitions  to  the  damage  axis 

In  employing  the  repetitive  computation  ap¬ 
proach,  the  probability  axis  partitions  did  not 
need  to  be  mapped  onto  the  damage  axis  in  the 
strict  sense  of  Section  3.3.  The  ordered  sample 
values  were  cumulatively  summed  and  nor¬ 
malized  onto  the  range  [0,  l].  Fig.  10  shows 
histograms  for  the  two  extreme  cases  (E  = 


0.5,  E  =  1.0)  determined  as  in  Fig.  9.  The  posi¬ 
tions  of  the  entries  most  closely  matching  each  of 
the  partition  values  were  determined  for  use  in 
the  next  step. 

5.5.  Find  conditional  expectations 

For  notational  convenience,  call  the  lower- 
damage  events  domain  I.  the  intermediate  events 
domain  II.  and  the  larger-damage  events  domain 
III.  Since  the  samples  were  ordered  from  least  to 
greatest,  the  conditional  expectations  were 
computed  by  summing  all  samples  between 
entries  corresponding  to  the  partition  values  and 
then  normalizing  the  sum  by  the  number  of 
entries  included  (see  equation  (2.14)).  The  un¬ 
conditional  expectations  were  similarly  Corn- 


cumulative  frequency 
of  occurence 


Fig,  10. 


r,  resource  damage 


A 
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puted  by  summing  all  the  samples  and  normaliz¬ 
ing  by  the  sample  size  (9000).  The  results  of 
these  computations  are  given  in  Tables  2  and  3 
for  the  extreme  event  and  pessimistic/optimistic 
partitions,  respectively:  the  trade-offs  are  des¬ 
cribed  in  Section  5.7.  The  conditional  and  un¬ 
conditional  expectations  are  plotted  against  the 
emission  levels  in  Figs,  11  and  12  The  points 
defining  /4(£)  in  Fig.  11  are  probably  erratic  due 
to  the  small  sample  size  (1 1-17  observations)  per 
point. 


5.6.  Generate  functional  relationships 

Since  the  emission  level  is  a  continuous  vari¬ 
able  with  E  €E  [0.  -*“*).  the  continuity  assumption 
(see  Section  3.5)  can  be  justified  for  linear  rela¬ 
tionships  except  for  fx{x )  in  Fig.  11  Table  4 
presents  coefficients  of  regression  for  relation¬ 
ships  of  the  form* 

/,(£)=  a  +  bE.  j  =  2.  3.4,  5,  (5.9) 


Table  2 

Extreme  event  partition 


Domain  1 


Domain  11  Domain  111  Unpartitioned 

(unconditional) 


-  £f  Tradeoff  A-  2*  Trade-off  Ap  ~  Trade-off  A u  2  Trade-off  A p 


1.0 

0 

0 

0.947 

1.80 

1.56 

2.09 

1.08 

0.704 

2.73 

1.26 

0  941 

1.00 

1.72 

1  47 

0  9 

10 

0.15 

0.871 

3.22 

5  22 

1.88 

1  93 

s 

r  l 

2.57 

2.25 

1.83 

0.9(H) 

3.08 

4  98 

o.s 

20 

0.60 

0.765 

5.76 

4.22 

1.73 

3.44 

5.25 

2.32 

4.02 

4.65 

0.810 

5.50 

8.78 

0.7 

30 

1  5 

0.667 

10.3 

12  3 

1.56 

6.15 

9.92 

2.13 

7.19 

-8.60 

0.708 

9.82 

11.8 

0.6 

40 

2.7 

0.569 

184 

196 

1.44 

11.0 

12.3 

->  ->7 

12.9 

7.25 

0.606 

17.6 

18.6 

0.5 

50 

4.8 

0.464 

32.9 

L- 

1  27 

19.7 

L+ 

1.99 

23.0 

L- 

0.496 

31.4 

L- 

Table  3 

Pessimistic/optimistic  partition 


'a 

7  w 
s 

i  ~ 

7  ^ 

=  S 

Domain  I 

Domain 

11 

Domain 

111 

(Unpartitioned 

(unconditional) 

o 

ZL 

Trade -off  A 

i: 

Trade-off  A 

n 

^  Trade-off  Au 

i  Trade-off  A 

i« 

/ 

'J 

/  — 

✓ 

— 

/  — 
a  7 
—  u 

/ 

J  'J 

'j  T 

y*  ^ 

£■ 

s 

s 

'J  O 

U  ^ 

X  ^ 

^  7 

1.0 

0 

0 

0.464 

3.38 

3.09 

1.19 

1.52 

1.26 

1  98 

1.10 

0.775 

1.00 

1.72 

1.47 

0.9(B) 

10 

0.15 

0.416 

6.03 

10.4 

1.07 

2.72 

4.21 

1  79 

1.97 

2.96 

0.9<K) 

3.08 

4  98 

0.8 

20 

0.60 

0.372 

10.8 

17.7 

0.959 

4.85 

7  56 

1.64 

3.53 

5.48 

0.810 

5.511 

8.78 

0.7 

30 

1.5 

0.322 

19.3 

21  2 

0.840 

8.68 

11.0 

1.47 

6.31 

9  46 

0.708 

9  82 

11.8 

0.6(A) 

40 

2.7 

0.265 

34.5 

34  7 

0.731 

15.5 

17.4 

1.35 

11.3 

12.8 

0.606 

17.6 

186 

0.5 

50 

4.8 

0.206 

616 

0.614 

27.7 

L+ 

1.19 

20.2 

L+ 

0.496 

31.4 

L+ 

J  ^ 
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E 


Fig.  11.  Exireme  event  conditional  expectations 
where 

a  =  intercept  on  damage  axis, 
b  =  slope  of  line  (damage  per  unit  of  emissions), 

KaE  +  b-f,)2 

R -  =  — 1 =  goodness  of  fit  , 
j  =  2,  3,  4,  5  , 


j  =  2, 3. 4, 5 

/;  =  l  ^  /)(£)  =  average  value  of  /,(£) , 

V  E 

j  =  2,  3.  4,  5  . 

The  cost  coefficients  are  discussed  in  Section 
5,7.1.  As  expected,  the  fits  are  quite  good  except 
for  that  of  extreme  event  domain  III,  fx(E). 


Table  4 

Coefficients  of  regression 


Curve 

a 

b 

c 

Rl 

Extreme  event: 
domain  I 

0.959 

-0  00891 

0.999 

domain  II 

1.60 

0.456 

- 

0.995 

domain  III 

1  37 

1.30 

- 

0  873 

Pessimistic/optimistic. 
domain  I  0  512 

-0.0433 

0.997 

domain  II 

1.14 

0.0456 

- 

1.00 

domain  III 

1.56 

0.397 

- 

0.997 

Cost 

5.81 

0  298 

0  344 

1.00 

5.7.  Employ  the  SWT  method 

To  determine  acceptable  risk  levels  in  the 
context  of  the  overall  decision  circumstances, 
functional  relationships  describing  other  decision 
factors  are  required  A  cost  function  is 
developed  and  trade-off  related  to  the  linear 
relationships  from  Section  5.6  are  found:  discrete 
trade-off  approximations  are  also  explored. 

5.7.1.  The  cost  function 

The  cost  of  sulphur  dioxide  (S02)  control  is 
given  in  Fig.  13.  This  analysis  employs  the 
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median  OTA  Cap  cost  estimates  between  the 
plant  cap',  which  assumes  any  plant's  emissions 
do  not  exceed  the  specified  limit,  and  the  least- 
cost  'state  cap',  where  required  'plant  cap 
reductions  are  achieved  through  intrastate  trad¬ 
ing  The  base  (current)  level  of  SO;  emissions  is 
taken  to  be  22.5  million  tons/year;  hence  £  =  0.5 
corresponds  to  a  reduction  of  SO;  emissions  to 
1 1  25  million  tons/year.  Median  costs  and  emis¬ 
sion  levels  for  each  of  the  eight  Cap  ranges  1.0  lb 
through  4  0  lb  are  given  in  Table  5. 

Given  that  the  curve  in  Fig.  13  is  approximately 
exponential,  coefficients  of  regression  for  a  rela¬ 
tionship  of  the  form 

/,(£)=  b  ea"-E)- c  (51°) 

are  shown  in  Table  4.  The  conditional  and  un¬ 
conditional  expectations  versus  additional  cost  of 
reduction  for  the  six  emission  levels  E  =  0.5 
through  £  =  1.0  are  plotted  in  Figs.  14  and  15  for 


Table  5 
Cost  data 


Emission  level 
(E) 

Additional  cosi 
($109) 

1.000 

0.000 

0.8982 

0.09780 

0.8436 

0.3542 

0,7939 

0.6410 

0.7281- 

1.227 

0  6449 

2.112 

0.5854 

2.875 

0.5431 

3  883 

the  extreme  event  and  pessimistic/optimistic 
partitions,  respectively. 


5.7.2.  Find  continuous  function  trade-offs 

Given  a  cost  function  (see  equation  (5.10))  of 
the  form  f\(E)=  Bea(K)-c  and  risk  functions 
(see  equation  (5.9))  of  the  form  /,(£)=  rE+s, 
j  =  2,  3, 4,  5,  trade-offs  may  be  obtained  for 


Additional  control  costs  ($  billion /yt,  1980  dollars) 
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SOURCE:  Office  of  Technology  AiMiimcnt.  1982 

Fig.  13. 


noninferior  points  on  the  Pareto-optimal  frontier 
(see  Section  2.4.3).  Since  £  is  a  scalar  policy 
variable,  an  explicit  equation  describing  the 
trade-offs  may  be  found  as  follows.  Solving 
equation  (5.9)  for  E  and  substituting  into  equa¬ 
tion  (5.10)  gives: 


f\  -  b  exp 


j  =  2.  3,  4,  5  . 


j  =  2.  3,  4.  5  . 

(5.12) 

Tables  2  and  3  list  the  trade-offs  associated  with 
each  emission  level,  range,  and  partitioning 
scheme.  Observe  that  A(/>0  for  all  £E 
(-x.x);  thus,  nonzero  trade-offs  for  E  =  1  are  to 
be  expected. 


A  --9£i 

"  <V, 


ab 


fr*** 


In  this  case. 


(5.11) 


5.7.3.  Find  discrete  trade-off  estimates 
The  trade-offs  may  also  be  estimated  using 
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Fig.  14.  Exireme  event  functional  space. 


equation  (3.7).  Tables  2  and  3  give  the  trade-offs 
for  a  decrease  of  the  emission  level  to  the  next 
smaller  value.  The  notation  L+  indicates  a  large 
positive  number. 

0. S.  Interpretation  of  results 

For  the  extreme-event  partitioning,  the 
decision-makers  are  reminded  that  besides  the 
lesser  damage  levels  presented  by  f<(E)  (the  un¬ 
conditional  expectation),  there  is  a  low.  albeit 
nonzero,  probability  of  the  major  damage  level 
presented  by  fA{x)  (higher  damages,  domain  III). 
Also,  reducing  E  causes  the  largest  magnitude  of 
reduction  in  domain  III  resource  damages  This 
information  is  not  available  from  the  un¬ 
conditional  expectation  alone. 

Alternatively,  through  the  pessimistic/opti¬ 
mistic  partitioning,  optimistic  decision-makers 
w'ould  tend  to  disregard  fA(E)  (domain  III);  thus 


they  would  spend  less  funds  for  emission  reduc¬ 
tion  by  making  their  expenditure  just  sufficient 
to  lower  fi(E)  to  an  acceptable  level.  Con¬ 
versely.  pessimistic  decision-makers  would  more 
seriously  consider  the  impact  of  /4(£)  and  spend 
more  funds  for  emission  reduction  to  lower  f*(E) 
to  an  acceptable  level  Note  that  the  cost  would 
be  manyfold  higher  to  reduce  f*{E)  to  a  parti¬ 
cular  damage  level  than  to  reduce  fi(E)  to  that 
same  level. 

Consider  policies  A  and  B  in  Fig.  15  and  Table 
3.  Policy  A  represents  a  relatively  high-cost/low- 
resource-damage  option  with  a  respectively  large 
trade-off:  S34.5  x  I0y/damage  unit  at  a  reduction 
level  of  40%  with  an  additional  cost  of  /.(*)  = 
$2.7  x  10y/vear.  For  this  policy,  the  expected 
resource  damage  is  f:(x)  =  0.265.  The  significance 
of  this  relatively  high  trade-off  is  that  a  small 
increase  in  the  level  of  resource  damage  results 
in  a  relatively  large  reduction  in  the  cost.  Policy 
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B.  on  the  other  hand,  is  an  example  of  a  rela¬ 
tively  low-cost/high-resource-damage  policy 
exhibiting  a  relatively  low  trade-off  value.  For 
this  10%  reduction  of  emissions,  the  additional 
cost  is  f\(x)  =  SO.  15  x  U)9/year,  the  expected 
resource  damage  is  f:(x)  =  0.416,  and  the  trade-off 

a  low  $6.03  x  I0g/damage  unit  This  means  that 
at  a  relatively  lower  cost,  the  expected  resource 
damage  can  be  markedly  decreased. 

Further  examination  of  Figs.  14  and  15  in¬ 
dicates  that  initial  reduction  in  expected  damage 
can  be  achieved  to  quite  a  significant  degree  with 
little  additional  cost.  Further  spending  has  the 
’must  dramatic  effect  on  the  reduction  of  domain 
ill  damages,  although  the  marginal  returns  for 
dollars  spent  are  less. 

Because  of  the  linear  nature  of  the  resource 
damage  versus  emission-level  relationship,  the 
damage  distribution  is  nearly  a  truncated  Gaus- 
Mun  with  little  discernible  skew.  Had  a  more 
distinctly  skewed  distribution  been  obtained. 


some  advantages  of  the  PMRM  might  have 
become  more  obvious.  The  resulting  risk  func¬ 
tions  were  linear,  as  might  be  anticipated  from 
the  linear  model,  however,  nonlinear  models 
may  not  yield  risk  functions  of  similar  nonlinear 
form 

In  brief,  this  example  quantified  the  risk 
objectives,  provided  a  multiobjective  framework 
for  risk  decisions  in  a  broader  context,  obtained 
trade-offs  between  risk  and  other  objectives,  and 
thereby  accommodated  decision-making  with 
explicit  and  quantitative  consideration  of  risk. 


6.  Evaluation  of  the  method  and  extensions 

6.  /  Evaluation  of  the  PMRM 

Fischhoff  et  al  [V\  suggested  seven  criteria 
against  which  a  risk-related  decision-making 
methodology  might  be  measured:  comprehen- 
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siveness.  logical  soundness:  practicality  in  rela¬ 
tion  to  real  problems,  people,  and  resource  con¬ 
straints;  openness  to  evaluation;  political  ac¬ 
ceptability;  compatibility  with  existing  in¬ 
stitutions;  and  conduciveness  to  learning  for 
future  risk  decisions. 

6.1.1.  Attributes 

In  view  of  these  seven  criteria,  the  following 
observations  can  be  made.  An  increase  in  the 
comprehensiveness  and  practicality  of  the  analy¬ 
sis  has  been  brought  about  by  presenting  in¬ 
formation  about  the  distribution  of  risk  in  a 
multiple-objective  format  with  other  objective 
functions,  allowing  risk  to  be  viewed  in  perspec¬ 
tive  to  other  important  criteria.  The  PMRM 
proceeds  in  a  structured  and  logical 
manner.  Explicit  steps  to  create  the  risk 
functions  and  the  trade-offs  provide 
an  openness  to  later  evaluation.  Transform¬ 
ing  the  risk  problem  into  a  multiple-objective 
problem  allows  the  decision-maker  to  consider 
the  political  acceptability  and  institutional  com¬ 
patibility  of  various  alternatives  through  multi¬ 
objective  decision-making  techniques  such  as  the 
surrogate  worth  trade-off  (SWT)  method. 
Finally,  the  structured  format  allows  each  well- 
documented  study  to  be  a  learning  experience 
for  improving  the  method  for  future  analyses. 

Beyond  the  seven  evaluation  criteria,  the 
PMRM  provides  some  additional  strong  points. 
Proper  development  of  the  risk  multiobjective 
separates  information  about  catastrophic  (pri¬ 
marily  low  probability)  events  and  low  damage 
(primarily  high  probability)  events,  thereby  cir¬ 
cumventing  a  point  of  contention  with  the  tradi¬ 
tional  expected  value;  thus,  more  information  is 
available  to  the  decision-maker  about  the  dis¬ 
tribution  of  risk.  Employing  the  SWT  method 
further  strengthens  the  PMRM  by  avoiding  the 
need  to  explicitly  assess  each  decision  maker's 
utility  function(s). 

6.1.2.  Sh  ortcom  ings 

This  section  describes  some  of  the  shortcom¬ 
ings  of  mathematical  decision-making  systems  in 
general  and  those  specific  to  the  PMRM. 

61.2.1.  Shortcomings  indigenous  to  mathematical 
decision-making  systems.  Several  difficulties  are 
shared  by  all  mathematical  decision-making  sys¬ 


tems.  The  comprehensiveness  of  the  analysis  is 
formulation  dependent;  that  is,  broad  and  clear 
formulation,  care  in  the  actual  decision-making, 
and  a  diligent  and  thorough  sensitivity  analysis 
provide  for  the  soundest  results.  Practicality  in 
terms  of  real  people,  problems,  and  resource 
constraints  requires  the  use  of  robust,  proven 
mathematical  models  of  the  risks,  other  objec¬ 
tives,  and  constraints  For  a  study  to  provide  the 
opportunity  of  learning  for  future  decisions, 
documentation  of  that  study  must  be  as  complete 
and  thorough  as  possible.  Finally,  the  alter¬ 
natives  included  in  any  analysis  must  be  struc¬ 
tured  in  such  a  way  that  compatibility  with  exis¬ 
ting  institutions  is  kept  in  mind. 

6  1.2.2.  PMRM  shortcomings.  There  are  some 
specific  shortcomings  indigenous  to  the  PMRM 
itself.  The  decision-makers’  utility  function(s)  are 
not  made  explicit  during  the  procedure,  so  the 
basis  for  the  decision  retains  some  subjectivity; 
however,  explicit  utility  functions  are  subject  to 
some  question  in  any  analysis.  The  basis  by 
which  the  probability  range  is  partitioned  could 
be  strengthened.  The  interpretation  of  the  risk 
functions  could  be  more  complete.  Although 
there  is  no  reason  to  doubt  their  solvability,  the 
PMRM  has  yet  to  be  applied  to  problems  in¬ 
volving  multidimensional  decision  and/or  risk 
vectors.  Partitioning  on  the  damage  axis  rather 
than  the  probability  axis  has  been  suggested,  but 
the  efficacy  and  practical  application  of  this 
option  has  vet  to  be  demonstrated 

The  simulator  approach  (Section  2.3)  requires 
a  large  amount  of  (computer)  calculation  and, 
therefore,  requires  either  easily  solvable  models 
for  the  riskv-loss  variables  and/or  computer 
packages  for  solution  approximations  of  multiple 
integrals.  On  the  brighter  side,  a  micro-  or  mini¬ 
computer  with  hard  disk  storage  capacity  should 
provide  adequate  computer  capacity  for  many 
problems. 

6.2.  Extensions 

The  PMRM  at  present  offers  exciting  and 
widespread  potential  use  in  risk-related  decision¬ 
making  problems,  and  several  suggestions  come 
to  mind  More  studies  employing  the  PMRM 
should  be  done,  particularly  examples  involving 
multidimensional  decision  and/or  damage  vec- 
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lors  and  partitioning  on  the  damage  (rather  than 
the  probability)  axis.  These  studies  could  also  be 
used  in  refining  the  interpretation  of  the  risk 
functions.  Development  of  a  more  theoretical 
basis  for  assigning  the  partitioning  ranges  could 
provide  for  a  better  communication  of  the  dis¬ 
tribution  of  risk  for  a  given  alternative.  Other 
theoretical  investigation  intended  to  find  an 
explicit  relationship  between  risk  function  trade¬ 
offs  similar  to  the  relation  among  risk  functions 
found  in  Section  4.3  could  be  quite  useful.  As 
may  be  recalled  from  Section  1.  Kaplan  and 
Garrick  [27]  suggested  a  frequency  analog  to  the 
probability  distribution  function  employed  by  the 
PMRM  Further  strengthening  of  the  theoretical 
basis  for  their  work  could  lead  to  the  develop¬ 
ment  of  the  PMRM  using  the  ‘frequency  dis¬ 
tribution  function  .  which,  in  turn,  could  prove 
quite  useful  in  the  policy-making  process 
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THE  SURROGATE  WORTH  TRADE-OFF  (SWT)  METHOO 
AND  ITS  EXTENSIONS 
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Abstract 


This  paper  briefly  summarizes  selected  published  results  on  the  Surrogate  Worth  Trade¬ 
off  (SWT)  Method--a  method  for  solving  multiple  objective  optimization  probl ems--and 
i  -  extensions.  The  development  of  the  SWT  method  is  briefly  discussed.  Theoretical 
casis  for  the  Kuhn-Tucker  multipliers  and  trade-off  functions  associated  with  Pareto 
optimal  solutions  is  presented.  The  SWT  method  is  then  extended  to  handling  multiple 
decision-makers.  The  Multiobjective  Statistical  Method  (MSM) ,  and  the  analysis  of  risk 
and  sensitivity  in  a  multiobjective  optimization  framework  using  the  SWT  method  are 
discussed  also.  A  case  study  in  water  resources  planning  with  several  noncommensurabl e 
objective  functions  is  summarized.  Finally,  special  attributes  of  the  SWT  method  are 
presented.  Secause  of  the  very  limited  scope  of  this  paper,  no  attempt  has  been  made 
to  relate  the  SWT  method  to  other  mul tiobjecti ve  optimization  methods. 
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subject  to  fj(.x)  -  Gj  »  j  f  i 


n;  x$X 


where 


3  mi  n  f j  ( x)  ,  x^X ,  j  3  1 , 2 , . .  .  ,  n 

ty  j  f  i,  j  3  are  maximum  tolerable  levels 


(3) 


such  as  e .  =  f.  +  £  • ,  and  e.  >  0. 

J  J  J  J 


The  equivalence  between  problems  (1)  and  (2)  is  proved  in  the  equivalence  theorem  by 
Haimes,  Lasdon  and  Wismer  [1971].  The  levels  of  satisfactory  z-  can  be  varied  para¬ 
metrically  to  evaluate  the  impact  on  the  single  objective  function  f^(x_).  Of  course, 
the  ith  objective,  f.(x),  can  be ‘replaced  by  the  jth  objective  fj(x),  and  the  solution 
procedure  repeated.  The  e-constraint  aporoach  facilitates  the  generation  of  noninfer¬ 
ior  solutions  as  well  as  the  trade-off  functions,  as  will  be  discussed  later. 

By  considering  one  objective  function  as  primary  and  all  others  at  minimum  satisfying 
levels  as  constraints,  the  Lagrange  multipliers  related  to  the  (n-1)  objectives  as 
constraints  will  be  zero  or  nonzero.  If  nonzero,  that  particular  constraint  does 
limit  the  optimum.  It  will  be  shown  that  nonzero  Lagrange  multipliers  correspond  to 
the  noninferior  set  of  solutions.  Furthermore,  the  set  of  nonzero  Lagrange  multipli¬ 
ers  represents  the  set  of  trade-off  ratios  between  the  principal  objective  and  each 
of  the  constraining  objectives,  respectively.  Clearly,  these  Lagrange  multipliers 
are  functions  of  the  optimal  level  attained  by  the  principal  objective  function,  as 
well  as  the  level  of  all  other  objectives  satisfied  as  equality  (binding)  constraints. 
Consequently,  these  Lagrange  multipliers  form  a  matrix  of  trade-off  functions. 

The  question  of  the  worth  ratios  still  remains  after  the  matrix  of  trade-off  functions 
has  been  computed.  The  worth  ratios  are  essentially  achieved  via  an  interaction  with 
the  decision-maker.  However,  since  the  worth  ratio  need  only  represent  relative  worth, 
no:  the  absolute  level  of  worth  of  the  objectives,  any  surrogate  ratio  which  varies 
mor.otonically  with  the  correct  one  will  suffice. 

1 . 3  The  Trade-off  punct^on 

The  following  development  shows  that  the  trade-off  functions  can  be  found  from  the 
values  of  the  dual  variables  associated  with  the  constraints  in  a  reformulated  problem. 
Reformulate  problem  (1)  as  follows: 


min  f,(x)  subject  to  xj-X,  f.(x)  5  e. , 

Y  1  J  J 


=  2,3 . n 


where  e  j  3  fj  +  c-,  z-  >  0,  j  -  2,3,. . . ,n 
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* 


varied  parametrically  in  the  process  of  constructing  the  trade-off  function. 

F c-mn  the  generalized  Lagrangian,  L,  the  system  (4): 

L  =  f](i)  +.I2XijCfj(x)  -  €j]  (5) 

where  j  *  2,3,...,n,  are  generalized  Lagrange  multipliers.  The  subscript  Ij  in  A 
demotes  that  A  is  the  Lagrange  multiplier  associated  (in  the  c-constraint  vector  opti¬ 
mization  problem)  with  the  jth  constraint,  where  the  objective  function  is  fj(xJ. 

A,.,  will  be  subsequently  generalized  to  associate  with  the  ith  objective  function  and 
*  •»  * 

the  jth  constraint,  \.j.  Denote  by  X  the  set  of  all  x,j ,  i  *  1,2,...,N,  and  by  ft  the 
set  of  all  j  *  2,3,.. .,n,  that  satisfy  the  Kuhn-Tucker  condition  for  problem  (5). 
The  conditions  of  interest  to  our  analysis  are 


X i j[f j (x)  -  £j]  =  0;  X^  >0;  j  3  2,3, ...,n  (6) 

Note  that  if  f .  ( x_)  <  e;  for  any  j  -  2,3,...,n  ( i .  e . ,  the  constraint  is  not  binding), 

J 

then  the  corresponding  optimal  Lagrange  multiplier  A ^  =  0. 

The  value  of  A^,  j  3  2,3,...,n,  corresponding  to  the  binding  constraints,  is  of  spe¬ 
cial  interest  since  it  indicates  the  marginal  benefit  (cost)  of  the  objective  function 

f  'x)  due  to  an  additional  unit  of  e<.  From  Eq.  (5).  assuming  that  the  solution  is  , 

J 

optimal,  the  following  results  can  be  derived1,  . 


5L 

3£j 


j  *  2,3, 


:;o:s ,  however •  that  for  xfX,  for  all  j 


Th'-S , 


f,(x)  -  L 


3f,(-) 

•  -  -IT- 


j  *  2,3, ...  ,n 


(7) 

(8) 

(9) 


In  the  derivation  of  the  trade-off  functions  in  the  SWT  method,  only  those  A^  >  0 
cor  responding  to  f j ( x)  =  Sj  are  of  interest  (since  they  correspond  to  the  noninferior 
solution).  Thus,  for  fj(x)  =  ,  Eq.  (9)  can  be  replaced  by  Eq.  (10): 


?M‘) 

xij(£j)  =  -  sfTFT 


(10) 


Clearly,  Eq.  (10)  can  be  generalized  where  the  index  of  performance  is  the  ith  object¬ 
ive  function  of  the  system  (1)  rather  than  the  first  one.  In  this  case,  the  index  i 
should  replace  the  index  1  in  \^y  yielding  A •  j .  Accordingly, 
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er.ce  band  is  assumed  to  exist  within  the  neighborhood  of  A*..  Additional  questions  to 

tte  CM  can  te  asked  in  the  neighborhood  of  A?.  to  improve  the  accuracy  of  A*,  and  the 
band  of  i nd i f f erence .  ^ 

There  are  three  ways  of  specifying  a  noninferior  solution  via  the  surrogate  worth 
function: 

1.  by  the  values  of  its  decision  variables, 

2.  by  the  trade-off  functions  A.,  j , . . .  **jn 

3.  by  its  objective  function  values  f,,...,f 

•  n 

Heocs,  we  can  have  W.^x, . xN)  or  WfJ(Xfl . A<n)  or  W.j(f1 . fj.  The  first 

is  generally  ruled  out  by  the  inefficiencies  of  decision  space  manipulations.  The 
second  may  suffer  from  problems  when  discontinuities  or  nonconvexities  occur  in  the 
functional  space,  but  can  be  used  in  other  problems.  The  third,  or  objective  function 
space,  approach  appears  to  be  best. 

As  an  example  of  how  the  method  works,  consider  a  three-objective  problem.  Several 

non  inferior  points,  ( f  3)Q . (VVk’  and  thei>  trade-offs,  <  >*  1 2  * x  1 3 )  0 . 

( *  1 2 * '  1 3 ^ k *  are  ^etermi ned ,  e.g.,  via  the  E -constraint  method.  The  decision-maker  is 
then  questioned  to  get  values  ’  *W13(  f2’Vk  and  W13(f2’f3)0,‘  ••• 

^13^2'^k*  can  shown  that  the  other  .  need  not  be  determined.)  Now,  since 
genera  1  ly  none  of  these  will  be  zero,  we  must  determine  more  noninferior  solutions  and 
their  trade-offs  than  before,  and  ask  more  questions  of  the  OM  until  we  find  an 

so  that  W12^2’^V*  and  W13^2*f3^*  doth  e(^ua^  zero. 

Since  the  worth  is  only  evaluated  at  known  noninferior  points,  it  is  guaranteed  that 
'f2,fV*  wl1'  riss  t0  a  feasible  solution  when  put  into  the  overall  mathematical 
motel.  The  same  guarantee  holds  when  W.^  (A.  y  . . .  ,A-n)  is  used. 

rthct  happens  if  there  can  not  be  found  a  pair  of  (f2,f3)*  whose  worth  functions  are 

both  zero?  In  that  case,  we  can  take  the  one  whose  worth  functions  are  closest  to 

zero  as  an  approximate  preferred  solution.  Note  that  the  noninferior  solutions  whose 

surrogate  worth  functions  are  all  zero  are  the  maximum  utility  solutions.  The  non¬ 
inferior  solution  whose  worth  functions  are  closest  to  zero  will  be  the  one  closest 
to  the  maximum  utility  solution. 

There  is  a  close  relation  between  the  surrogate  worth  function,  W.  and  the  partial 
derivatives  of  the  utility  function. 

• n  m.ul  ti  objective  analysis  it  is  assured  implicitly  that  the  decision-maker  maximizes 
nis  utility,  which  is  a  monotonic  decreasing  function  of  the  various  objective 
functions . 


"he  relationship  between  the  surrogate  worth  function  and  the  utility  function  is 
presented  elsewhere. 
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Finally,  one  rr.ay  question  whether  an  interaction  with  the  DM  in  the  function  space 
should  always  yield  a  W'^A^)  =  0,  i.e.,  an  indifference  solution.  Two  cases  may  be 
i certified  here. 

1.  The  DM ' s  response  is  always  on  one  side  of  the  scale  for  all  A^  corresponding 
to  the  Pareto  optimum  solutions.  That  is  to  say,  the  DM's  answers  are  either  all 
on  the  positive  or  all  on  the  negative  scale  of  W^.  This  really  means  that  the 
CM  is  always  willing  to  improve--say  objective  l--at  the  expense  of  degrading  ob¬ 
jective  2  in  the  entire  Pareto  optimal  space.  This  case,  while  it  may  actually 
happen,  is  of  no  particular  interest  here,  since  it  reduces  the  multiobjective 
problem  to  a  sir.gl e-objecti ve  optimization  problem. 

2.  Should  the  DM's  response  in  the  function  space  be  on  the  positive  scale  of  for 

sore  values  of  find  negative  for  other  sets  of  values  of  A^*  then  (assuming 

consistency  in  the  DM  response  and  continuity  in  A^)*  it  can  be  guaranteed  that 

a  value  of  2  0  exists  which  corresponds  to  an  indifference  solution  with  A^> 
i.e.  W12(.X*2)  =  0. 

1 , 5  Transforms ti on  to  the  Decision  Space 

O^re  the  indifference  bands  have  been  determined  for  A* ^ ,  the  next  and  final  step  in 
tr.e  SWT  method  is  to  determine  an  x*  that  corresponds  to  all  A*^.  To  each  A*  deter¬ 
mined  from  the  surrogate  worth  function  via  the  interaction  with  the  decision-maker 
there  corresponds  fT(x),  j  =  l,2,...,n,  j  t  i.  These  fT(xJ  are  the  values  of  the 

j  j 

functions  fj(x)  at  the  equality  constraints  Cj  so  that  A*j[fT(x}  -  Cj J  r  0.  Accord¬ 
ingly,  the  optimal  vector  of  decisions,  x* ,  can  be  obtained  by  simply  solving  the 
following  optimization  problem: 

min  f‘x)  subject  to  f.(x)  5  fT(x)  j  =  l,2,...,n;  j  f  i  (14) 

xtX  J  J 

""re  system  Eq.  (14)  is  a  common  optimization  problem  with  a  single  objective  function. 
Tqe  solution  of  Eq.  (14)  yields  the  desired  x*  for  the  total  vector  optimization  prob- 
1 en  posed  by  Eq .  ( 1 ; . 

T!r e  consistency  of  :.he  DM  should  not  always  be  assumed.  The  DM  may  show  nonrational 

bena.’icr  or  provide  misinformation  at  times.  The  SWT  method  safeguards  against  this 

by  crosscneckir.g  tr.e  resulting  AT..  It  has  been  shown  elsewhere  that  one  set  of 

^  J  r  9 1 

A '  1  in  solving  the  r.ul  tiobjecti  ve  problem  posed  previously. u  J 

! :  is  always  possible,  however,  to  generate,  for  example,  A*^,  /  -  3 .  find  ^*3  (V1’a  an 

ir -redaction  with  tr.e  DM).  and  to  check  that  indeed  the  following  relation  holds: 

>r3  =  satisfies  the  general  relationship  A.j  *  A.^A^. ,  for  i  f  j, 

A.- j  G;  i ,  j  s  1,2,...  , n ) . 
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Theorem  1 :  Given  the  mul tiobjecti ve  problem  (1)  referred  to  as 
Problem  1,  then  for  the  feasible  set  of  \*.t  there  exists  a  corresponding 
feasible  set  of  desicions  x*. 

Proof:  Rewrite  Problem  1  as  follows: 


min  f.(x) 

xa  1 


1  \.f. 

jn  ij  j 


If  all  f^(xj»  k  =  lf2,...,n,  are  continuous  and  the  solution  set  X  is  compact,  then 
this  problem  must  have  a  solution  (by  Weierstrass'  Theorem). 


These  assumptions  are  very  mild.  Compactness  of  X  can  be  guaranteed  by  imposing  fin¬ 
ite  upper  and  lower  bounds  on  each  component  of  the  decision  vector  x_,  assuming  the 
constraint  functions  g^(xj  are  continuous.  Continuity  assumption  of  all  fj(x)  and 
g^(xj  (as  defined  in  Eq.  1)  is  common  in  mathematical  programming. 

Let  x_*  be  a  solution  for  a  given  Xt..  Then  AV's  are  the  optical  trade-off  values 
(Lagrange  multipliers)  for  the  problem: 


min  f.(xj  subject  to  the  constraints  f.(x)  <  f.(x*)  =  e . 

xa  1  j  -  j  -  j 

j  =  l,2,...,n;  i  f  j 

Thus,  x_*  is  in  X  and  A*^.  's  are  the  desired  Lagrance  multipliers. 

The  feasibility  of  a  solution  x*  corresponding  to  AT.  can  also  be  shown  on  the  basis 

“fill  1 J 

of  the  Lambda  Theorem  by  Everett.1  J 

It  is  helpful  to  summarize  the  three  major  steps  in  the  SWT  method.  These  are: 

Step  1.  Identify  and  generate  noninferior  (Pareto  optimum)  solutions,  along 

with  the  trade-off  functions,  A^,  between  any  two  objectives  functions 
f j  (x.)  and  fj(x),  i  f  j.  It  can  be  shown  that  under  certain  mild 
conditions,  one  set  of  (n-1)  trade-off  functions,  *  j  2  *  *  *  *  ’  ^*  1  n  ’  w1^ 
suffice  to  generate  all  other  X.y  i  f  j ,  i ;  j  =  l,2,...,n. 

Step  2.  Interact  with  the  DM  to  assess  the  indifference  band  wnere  the  surro¬ 
gate  worth  function  W^.  ;(A*j)  *  0.  It  was  shown  tnat  under  certain 
mild  conditions  W .  ^  depends  only  on  \^y 

Step  3.  Determine  the  optimal  decision  set,  x_*,  using  the  optimal  trade-off 
values  A* j  . 
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2.  Trade-Off  Interpretation  of  Kuhn-Tucker  Multipliers 

For  convenience,  we  redefine  problems  (l)-(3)  in  this  section.  J  Let 

g^x)  S  0,  i  *  1 •  •  .m) 

wnere  g,  is  a  continuously  differentiable  real-valued  function  defined  on  RN.  Oefine 
Pk(e)  as: 

min  fk(x)  subject  to  fj(x)  s  Cj,  j  =  1 . n,  j  /  k  j  p  ^ 

g.(x)  5  0,  i  *  1 ,. . .  ,m,  and  xeR'  1 

i  n/\  ,*  4.u  \  *  n  i  ^  k  3fld  U-.  i  s  1  *  .  •  •  ^  Optima  1 

Let  x_*  solve  Pk(e)  W1 *kj *  J  ’  1  ’ ’ *  * *n’  J  f 

Kuhn-Tucker  multipliers  associated  with  fj's  and  g^'s  constraints,  respectively. 

(‘)ote:  \*  •  and  u*  are  the  optimal  Kuhn-Tucker  multipliers  of  Pk(o)  'f  [  —  *  •  •  * » 

.  ■  „  ,.*)  satisfies  the  Kuhn-Tucker  conditions  for  optimality  for 

' <k-t’/Tck+l  ’ '  ” '  ’"n’  1  m 

Pk(e).) 

The  key  concept  underlying  relationships  between  Kuhn-Tucker' multi  pliers  XJj  and 
trade-offs  between  ffc  and  f.  at  x*  is  the  sensitivity  interpretation  of  the  multipliers. 

For  convenience,  we  shall  restate  Luenberger's  Sensitivity  Theorem  using  notations 

[13] 

suitable  for  our  discussions. 

Sensi t i v i tv  Theorem 

Given  t'z'i ,  ,  let  x*  solve  PkU5)  with  X£-.  j  f  k,  being  the  corresponding  Kuhn-Tucker 
multipliers  associated  with  the  constraints  fj(x)  5  e*.  j  f  k.  If 

i)  x*  is  a  regular  point  of  the  constraint  of  Pk(o  ) 

ii)  the  second-order  sufficiency  conditions  are  satisfied  at  x  *  >  and 

iii)  there  are  no  degenerate  constraints  at  x*,  then 

=  -3fk(x*)/3Gj  for  all  j  / 

Corel  1  a r/  1 

X*  solves  Pk(e°)  and  satisfies  (i),  (ii).  and  (iii),  then  there  exists  a  neighbor¬ 
hood  N(e°)  of  e°  so  that,  for  all  ckK(e°).  x(e),  which  uniquely  solves  Pk(e),  locally 
exists,  and  is  continuously  differentiable  function  of  e  with  x(c°)  =  £*. 

Corollary  2 

„'ith  all  the  hypotheses  of  the  sensitivity  theorem  satisfied,  there  exists  a  neighbor- 
rood  N(c:)  of  c°  so  that  for  each  j  such  that  >  0,  fj(x(s))  =  £j  for  all  ekN(e°). 

7-e  results  in  the  above  two  corollaries  arise  naturally  from  the  proof  of  the  sensi¬ 
tivity  theorem  which,  in  turn,  relies  heavily  on  the  implicit  function  theorem.  It 
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should  be  stressed  here  that  conditions  (i).  (ii).  and  (iii)  required  for  the  sensi¬ 
tivity  interpretation  of  Kuhn-Tucker  multipliers  above  are  merely  sufficient 


Condi ti on 


„  Cl*] 


3.  The  SWT  Method  With  Multiple  Decision-Makers 

Often  systems ,  which  are  best  modeled  by  multiple  objective  functions,  are  also  char¬ 
acterized  by  more  than  one  decision-maker  (DM).  Most  public  policy  issues,  for  example, 
lend  themselves  to  multiple  objectives  and  multiple  DMs.  A  hierarchy  of  DMs  often 
exists  within  an  agency  or  corporation,  and  preferred  noninferior  solutions  are  trans¬ 
ferred  from  one  level  of  the  hierarchy  to  another.  The  SWT  methods  very  well  suited 
to  incorporate  multiple  DMs  and  their  corresponding  preferences." 

In  the  case  study  discussed  in  section  6,  for  example,  the  Planning  Board  of  the  Maumee 
,iver  Basin  consists  of  eight  members  from  federal,  state,  and  regional  agencies.  The 
g„ard  is  in  charge  of  developing  a  basin-wide  comprehensive  plan  that  is  responsive 
t3  environmental,  economic,  social,  legal,  political,  and  institutional  needs.  However, 
each  member  of  the  Board,  as  a  decision-maker,  exercises  his  mandate  to  be  responsive 
simultaneously  to  his  professional  judgment,  his  agency's  stand,  and  the  public  pre¬ 
ferences  as  voiced  by  various  public  hearings  and  other  media.  Clearly,  in  applying 
-/surrogate  Worth  Trade-off  method,  different  indifference  bands  may  result  by  inter¬ 
acting  with  each  Planning  Board  member.  The  key  question  is  how  would  the  SWT  method 
be  rnodified  to  handle  this  situation. 

Three  major  cases  of  multiobjective  optimization  problems  with  multiple  decision- 
makers  have  been  discussed  in  the  literature. ^  These  are  the  direct  group  decision¬ 
making  systems,  the  representative  decision-making  systems,  and  the  political  decision 
simulations.  These  three  classifications  will  not  be  discussed  here,  but  rather  a 
more  general  case  will  be  assured  here  for  simplicity. 

Consider  the  multicbjective  optimization  problem  posed  by  Eq.  0).  where  an  interaction 
with  tr.e  DMs  for  assessing  the  corresoonding  trade-offs  and  preferences  that  lead  to 
*  0  takes  place.  Two  cases  will  be  identified  here-the  ideal  and  the  probab  e. 

Tng  Id-al  Case:  In  assessing  the  trade-offs  and  preferences  with  the  DMs  it  is  as-^ 
sumed  in  the  ideal  case  that  the  indifference  bands  generated  by  all  the  DMs  for  al 
w  j  }  j  s  2  ,n,  have  a  common  indifference  band,  £,  as  depicted  in  ig. 

:’i;  situation  is  unlikely  to  happen;  however,  it  provides  a  medium  for  understanding 


fro  probable  case. 


All  the  indifference  bands  in  Fig.  2  correspond,  of  course,  to 
w/  I'  o" however!  they  are  plotted  at  different  levels  on  the  U,j  scale  in  order  to 
distinguish  among  tne  indifference  bands  of  the  various  DMs. 

T'-o  urobabl*  Cas=:  In  the  probable  case,  no  common  indifference  band  can  be  found 
-“TThe  DM.s Tn.is  case  is  depicted  in  Fig.  3.  The  Surrogate  Worth  Trade-off 


97 


f* 


rethod  provides  an  explicit  and  quanti tati ve  mechanism  in  simulating  the  decision- 
rakers'  preferences  with  respect  to  the  trade-offs  between  any  two  objective  functions 
t -erti fy ing  the  differences  in  the  OMs 1  preferences  is  a  first  step  in  closing  these 
c=ps  through  the  inevitable  process  of  negotiation  and  compromise.  These  negotiations 
rr>y  take  different  forms  and  are  expected  to  lead  to  an  agreeable  decision  (depending 
cn  the  rules  of  the  game  whether  a  simple  majority*  absolute  majority,  consensus,  or 
other  is  needed  for  an  agreed-upon  decision). 


4  Multiobjective  Statistical  Method  (HSM)_ 

Tr.e  Multiobjective  Statistical  Method  (MSM)  has  been  developed  to  integrate  statistica 
attributes  with  ir.ul  tiobjecti  ve  optimization  methodologies,  such  as  the  SWT  method. 

Outlines  of  the  KSM  are  presented  herein: 

Lat  x  denote  a  vector  of  decision  variables  and  f(y}  a  vector  of  noncommensurable  ob¬ 
jective  functions  with  i  the  state  vector  of  the  system  being  analyzed.  The  state 
vector  is  given  by  £  *  l(x;r)  where  r  is  a  vector  of  random  variables  modelling  the 
uncertainties  in  the  system.  Let 

-  *  (rl,r2 . rn^ 

w-ere  each  of  the  r-,  i  *  l,2,...,n  are  real-valued  random  variables  satisfying 
Hi  *  r.  <  rr  i  -  1,2 . n. 


Let 


Aij  •  j  *  1,2 . Ji 

be  a  collection  of  subsets  of  [r.,r.]  with  probabilities  P(A.jt),  i  =  1,2... 
used  for  a  given  study.  Define  the  n-fold  probabil i ties 


. ,n  to  be 


P=  ■  *!J2 


•  -  ‘"J.1 


w-are  1  <  j.  <  J  ^ ;  i  =  l,2,...,n  and  m  is  an  index  variable  from  the  set 

n 

{1,2 . M} ,  M  =  tt  J.. 

i  =  l 

Here,  p  is  the  probability  that 


rlcAlj1’  r2cA2j2 . rn£AnV 


T“2  planning  of  the  system  is 


formulated  as  the  multiobjective  optimization  problem 
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min  _f(yj 
X6X 


fL\i) 


s.t.  G(x)  <  0 


where  y  *  ^(x;r_)  is  a  vector  of  random  variables. 

For  £eX,  determine  /^(i.r^) ,  where  is  the  vector  of  random  variables  with  probabi¬ 
lity  b  for  m  =  1 ,2, . . . ,M.  The  most  convenient  way  to  define  the  sets  A.,  is  by  dis- 
J  rnu 

cretizing  the  intervals  [r^.r^]  into  J.  subintervals  such  that 

Ari  =  — -  and  A.^  =  (r^  +  (j-l)6ri ,jAr.]. 

The  components  of  are  taken  to  be  the  midpoints  of  the  appropriate  subintervals  of 
[_r .  ,r . ]  dictated  by  the  index  variable  m. 

To  account  for  the  random  nature  of  the  state  vector  jr  in  the  optimization  problem, 
define  the  quantities  f^(yj  =  E C f ^ (^f) }  where  i  -  1,2,...,L  and  £-.•}  denotes  expecta¬ 
tion.  Then 

f£(i)  ■  --  J1  ^(y(£;rm))pm 

where  _x6X  represents  a  fixed  set  of  decisions. 

In  order  to  proceed  with  the  optimization  problem,  it  is  necessary  to  obtain  a  func¬ 
tional  relationship  which  maps  _x  into  f^(x.)  for  each  objective  function,  l  =  l,2,...,l. 
One  possible  method  is  to  obtain  a  collection  of  ordered  pairs  {x,f„(x)}  and  use  a 
curve-fitting  technique  to  determine  the  functional  relationships  f£(*)»  for  each  l . 


The  problem  has  now  been  reduced  to 

min  ?( x) 
xCX 

s.t.  G(x)  <  9 

which  is  a  deterministic  multiobjective  optimization  problem  and  can  be  solved  by 
standard  procedures,  such  as,  the  Surrogate  Worth  "’’rade-off  ^SWT)  method  to  determine 
the  "optimal"  decision  vector  £*\ 


5.  Risk  and  Sensitivity  As  Multiple  Objective  Functions 

At  present,  most  mathematical  models  treat  important  system  characteristics  such  as 
risk,  uncertainty,  sensitivity,  stability,  responsi vi ty ,  i rreversibi li ty,  etc.,  either 
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by  r.rsns  of  system  constraints  or  by  artificially  imbedding  them  in  the  overall  index 
of  performance.'-18-'  The  systems  analyst  (the  modeler)  assumes  both  the  role  of  the 
professional  analyst  and  the  decision-maker  by  explicitly  or  implicitly  assigning 
^eights  to  these  and  other  noncommensura te  system  characteristics,  thus  commensurating 
them  into  the  performance  index  (the  mathematical  model’s  objective  function). 
Obvic-isly*  this  process  is  questionable  even  where  the  analyst  is  the  decision-maker. 
*r.en  he  is  not,  the  result  will  seldom  be  the  decision-maker’s  optimum. 


It  is  argued  elsewhere  that  the  above  system  characteristics  can  and  should  be  quanti¬ 
fied  and  included  in  the  mathematical  models  as  separate  objective  functions.'-'9-' 

These  should  then  be  optimized  alorg  with  tne  original  model's  objective  function 
(index  of  performance),  to  allow  the  deci sion-maker(s)  to  select  a  preferred  policy 
(solution)  from  within  the  Pareto  optimal  set.  Any  procedure  short  of  recognizing 
t^ese  characteristics  as  objective  functions  in  their  own  right  essentially  compro¬ 
mises  tne  modeling  process. 


Ir.  previous  work  a  number  of  questions  associated  with  risk  and  uncertainty  have  been 
tentatively  explored  to  stimulate  further  analysis  and  research  into  the  quantifica¬ 
tions  of  these  factors  for  use  in  multiobjective  optimization  analysis."20"  A  great 
-any  problems  exist  in  water  resources  systems  and  other  civil  systems  involving 
resources  in  which  avoidance  of  risk  and  uncertainty  is  often  in  fact  the  dominating 
objective.  If  suitable  quantitative  measures  of  these  objectives  can  be  formulated, 
tr.en  the  Surrogate  Worth  Trade-off  method  or  other  multiobject  *e  optimization  method¬ 
ologies  .can  determine  the  optimal,  or  at  least  superior,  combinations  of  risk  and 
tcricus  forms  of  return. 


This  somewhat  preliminary  analysis  and  discussion  indicates  that  quantitative  measures 
of  risk  can  be  defined  and  utilized  as  objectives  to  be  optimized  in  a  multiobjective 
control.  In  some  instances  even  uncertainty  (no  probability  distribution  data)  can 
be  treated  adequately. 

An  indication,  however,  is  not  an  accomplished  fact  and  much  insight  and  analysis  will 
oe  required  to  quantify  the  major  risk  factors  involved  in  common  water  resources 
systems  well  enough  to  include  them  in  multiobjective  decision  analysis. 

7r :  p-eposed  consideration  of  risk  in  a  multiobjective  framework  might  be  used  system¬ 
atically  to'-21*': 

(i)  Assist  planners,  professionals  and  decision-makers  involved  in  resources 
planning  and  management  in  general  and  in  water  and  related  land  resources 
in  particular. 

(ii)  Quantify  and  display  the  trade-offs  involved  in  reducing  risk,  sensitivity, 
irreversibility  and  other  systems  characteristics  (viewed  as  systems 
objectives)  along  with  reducing  cost  or  other  performance  indices,  where 


$  *  * 


100 

all  these  objectives  are  kept  in  their  noncommensurable  units. 

(iii)  Insure  comprehensive  consideration  of  economic  issues,  social  well-being, 
health  hazards,  environmental  issues,  irreversible  impacts,  and  other 
costs  and  benefits  regardless  of  commensurabi 1 i ty  through  the  use  of  the 
multiobjective  approach. 

(iv)  Reduce  the  uncertainty  surrounding  resources  planning  and  development 
in  general,  and  water  and  land  resource  development  and  management 
decisions  in  particular,  by  the  development  of  more  realistic  systems 
models,  and  accurately  displaying  the  probable  consequences  of  the 
decisions  and  policies  followed. 

6.  Maumee  Study:  Case  Study  for  Multiobjective  Optimization 

The  Maumee  study  addresses  itself  to  the  systematic  evaluation  and  formulation  of  a 
planning  and  management  framework  for  solving  the  critical  problems  of  water  and  land 
resources  in  the  Maumee  River  8asin.  The  planning  process  has  been  carried  out  in 
complete  cooperation  with  the  Maumee  River  8asin  Level-8  Planning  Board.  Results  and 
findings  of  this  study  are  utilized  in  the  Level-B  planning  effort,  and  in  formulating 

r  22 1 

the  recommended  plan  for  the  Basin. L  J 

While  a  great  many  studies  related  to  water  and  land  resources  planning  were  conducted 
in  the  past,  only  a  handful  have  been  sufficiently  comprehensive  in  nature  to  integrate 
all  the  inherently  complex,  and  interacting  components  in  a  multiobjective  framework. 
rhe  problems  of  land  and  water  resources  of  the  Maumee  Basin,  in  particular,  serve  to 
enohasize  the  need  for  such  an  Integra tion-- if  the  consequences  in  the  future  are  to 
be  adequately  considered  and  planned  for. 

To  facilitate  the  process  of  comprehensive  planning  in  the  Maumee  Basin,  a  hierarchical - 
mul ticbjec ti ve  modelling  and  optimization  structure  was  developed;  and,  based  on  this 
structure,  a  computer  program  was  developed  to  generate  alternative  plans,  and  the 
associated  trade-offs  using  the  Surrogate  Worth  Trade-off  methodology.  The  seven 
major  considerations  regarding  land  and  water  resource  management  in  the  8asin--water 
quality,  water  supply,  protection  of  agricultural  land,  sedimentation,  flooding,  out¬ 
door  recreation,  and  fish  and  wi Idl ife--have  been  converted  to  a  set  of  specified 
objectives  for  rnul ti objective  analysis. 

Linear  models  were  developed  for  each  of  these  objectives.  Using  these  models,  the 
level  of  objectives  among  the  various  components  have  been  generated  for  a  range  of 
feasible  alternative  solutions:  minimum  economic  development,  minimum  environmental 
quality,  economic  development  and  environmental  quality  alternatives  used  were  the 
alternatives  developed  by  the  Maumee  Planning  Board  and  presented  by  The  Great  Lakes 
Basin  Commission  at  public  forums  held  January,  1976. 


Trade-off  values  were  determined  for  each  of  these  alternative  scenarios.  For  example, 
under  environmental  quality,  planning  subarea  1,  the  worth  of  reducing  an  additional 
ton  of  sheet  erosion  on  agricultural  land  is  $3.4,  given  a  45%  reduction  of  the  pro¬ 
jected  sheet  erosion  in  the  year  1990  has  already  been  achieved.  These  trade-off 
values  with  their  respective  level  of  achieved  objectives  are  presented  by  planning 
subarea  for  each  component  and  each  alternative  scenario  in  Table  1. 

Planning  Board  members  were  then  asked  to  specify  their  range  of  indifference  with 
regar'd  to  economic  development  (cost)  and  environmental  quality  (unit  of  objective 
acnievement)  trade-offs.  The  overlapping  range  of  indifferences  obtained  represents 
the  optimal  solution  area  along  the  trade-off  curve,  i.e.  the  consensus. 

The  range  of  consensus  by  component  can  serve  as  one  of  the  guidelines  for  formulation 
of  the  Selected  Plan.  In  this  case,  the  Selected  Plan  used,  which  was  the  one  arrived 
at  by  the  Maumee  Planning  Board  and  Citizen's  Advisory  Committee,  coincides  with  the 
overlapping  range  of  indifferences  obtained  by  the  SWT  method.  Trade-offs  and  per¬ 
centage  level  reduction  in  gross  needs  corresponding  to  this  plan  were  subsequently 
generated,  and  are  also  shown  in  Table  1. 

The  trade-offs  can  also  be  evaluated  by  a  comparison  of  planning  subareas.  The  five 
planning  subareas  in  the  Maumee  River  Basin  have  different  hydrology,  geography, 
economic  charac teris tics,  population  density,  and  other  features.  Consequently,  the 
overall  Basinwide  objective  would  be  best  achieved  by  capitalizing  on  these  local  and 
subarea  attributes.  For  example,  under  the  Selected  Plan,  a  reduction  in  sheet  erosion 
of  agricultural  land  would  be  achieved  at  45  percent  of  the  1990  projected  gross  ero¬ 
sion  for  both  PSAs  1  and  3.  Yet  the  marginal  cost  values  of  reducing  an  additional 
ton  of  sheet  erosion  are  $3.40  per  ton,  and  $8.20  per  ton  for  PSAs  1  and  3, 
respectively.  These  marginal  cost  values  are  the  trade-offs  generated  by  the  Surrogate 
Worth  Trade-off  method  as  given  in  Table  1.  In  other  words,  in  the  recommended  plan 
of  sheet  erosion  reduction,  an  additional  reduction  of  sheet  erosion  in  PSA  1  would 
tost  less  than  half  of  that  of  PSA  3.  Therefore,  from  a  Basinwide  point  of  view, 
simple  economics  dictates  that  an  additional  sheet  erosion  reduction  in  PSA  1  should 
be  pursued  at  the  expense  of  a  lower  reduction  in  PSA  3  while  still  maintaining  the 
total  net  reduction.  However,  the  institutional  arrangements  are  not  available  for 
that  purpose.  Today,  there  is  no  system  of  incentives,  rebates,  taxation,  or  other 
mechanisms  that  induces  the  various  sectors  of  the  public  in  the  Basin  to  be  respons¬ 
ive  to  the  ecological,  hydrologi cal  ,  or  environmental  needs  beyond  the  level  of  com¬ 
pliance  with  the  law.  Other  examples  that  could  be  mentioned  are  ground  water  man¬ 
agement  (pumping  and  recharge)  and  waste-water  treatment  (level  of  treatment  and 
effluent  discharge  in  a  specific  reach). 
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7.  Summa ry 

The  Surrogate  Worth  Trade-off  (SWT)  method  can  be  used  to  analyze  and  optimize  multi - 
objective  optimization  problems.  A  brief  summary  of  the  SWT  method  is  presented 
here.^ 

1.  The  SWT  method  is  capable  of  generating  all  needed  noninferior  solutions 
to  a  vector  optimization  problem. 

2.  The  method  generates  the  trade-offs  between  any  two  objective  functions 
on  the  basis  of  duality  theory  in  nonlinear  programing.  The  trade-off 
function  between  the  i th  and  jth  objective  functions,  \.y  is  explicitly 
evaluated  and  is  equivalent  to 

3.  The  deci si  on-maker  interacts  with  the  systems  analyst  and  the  mathematical 
model  at  a  general  and  very  moderate  level.  This  interaction  is  accom¬ 
plished  by  the  generation  of  the  Surrogate  Worth  functions,  which  relate 
the  decision-maker's  preferences  to  the  noninferior  solutions  through  the 
trade-off  functions.  These  preferences  are  constructed  in  objective 
function  space  (more  familiar  and  meaningful  to  the  decision-makers)  and 
only  then  transferred  to  the  decision  space.  This  aspect  is  particularly 
important,  since  the  dimensional i ty  of  the  objective  ^unction  space  is 
often  smaller  than  that  of  the  decision  space.  The  preferences  yield  to 
an  indifference  band,  where  the  deci sion-maker  is  indifferent  to  any 
further  trade-off  among  the  objectives. 

A.  The  SWT  method  provides  for  the  quantitative  analysis  of  noncommensurable 
objective  functions. 

5.  The  method  is  very  well  suited  for  the  analysis  and  optimization  of  multi- 

r  24  T 

objective  functions  with  multiple  decision-makers.L  J 

5.  The  method  possesses  an  appreciable  computational  efficiency,  and  has  some 
advantages  over  other  existing  methods,  when  the  number  of  objective 

r  2  s  i 

functions  is  three  or  more.'  J 

7.  The  method  is  applicable  to  both  static  (mathematical  p’-ogramni ng)  and  dynamic 
(optimal  control)  vector  optimization  probl ems . 

3.  The  method  has  been  extended  to  incorporate  an  interactive  mode  between  the 

f  27 1 

analyst  and  the  QMs--!nteracti ve  Surrogate  Worth  Trade-Cff  ( I SWT )  method.1  J 

9.  The  method  has  been  extended  to  incorporate  statistical  data--Mul tiobjecti ve 
Statistical  Method  (,MSM).^^ 
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Figure  1. 


Figure 


Determination  of  the  indifference  band  at  A* 


Common  indifference  band  in  the  ideal  case 
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Figure  3. 


Indifference  bands  in  the  probable  case. 
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Table  1. 


Levels  of  Objectives  in  Percentage  of  Project  Gross  Needs  for 
the  year  1990,  the  Associated  Trade-offs,  and  land  Management 
Costs. 


s. 


oyiiCTivs 

PSA 

PERCENT 

LEVEL 

TRADE¬ 

OFF 

R^J'jccior.  in 

1 

45 

3.4 

''TESi. 

TrSTTan 

2 

37 

4.6 

( •*:.  UnJ) 

3 

46 

>.2 

ftjna/yr.) 

4 

57 

5.4 

5 

19 

7.3 

Retention 

1 

13 

1.3 

i Threat  .on 

Ja/s) 

l 

J  7 

2.6 

3 

18 

3.2 

4 

27 

3.3 

5 

19 

3.2 

hi M’  i  fe 

1 

74 

3.0 

i-ier  Jays) 

2 

S4 

2.5 

3 

1W 

2.3 

4 

87 

2.4 

S 

20 

3.0 

Hoosl  PUin 

1 

7.0 

67 

iVw.mi iit.on  .  . 

(acres) 

2 

5.0 

49 

3 

8.0 

49 

4 

5.0 

49 

5 

3.0 

51 

Total  LanJ 

1 

22,949.000 

Cost   

T TT - 

2 

35,048, 

,000 

3 

35, SSS, 

,000 

4 

54,003,000 

5 

11,670,000 

jg  KIN  £3 


PGRONT 

LEVEL 

TRADE¬ 

OFF 

PERCENT 

LEVEL 

TRADE* 

OFF 

28 

3.1 

11 

2.8 

30 

4.5 

29 

4.5 

31 

8.2 

26 

8.2 

28 

5.4 

1.1 

S.4 

Increase 

18 

6.9 

18 

6.  S 

15 

1.3 

17 

1.3 

28 

2.6 

29 

2.6 

19 

3.2 

20 

3.2 

28 

3.3 

2t 

3.3 

19 

3.3 

19 

3.3 

66 

3.0 

52 

2.9 

64 

2.5 

65 

2.5 

144 

2.5 

1S4 

2. 5 

84 

2.4 

77 

2. 5 

20 

1.7 

11 

1.7 

12 

67 

23 

67 

11 

49 

21 

49 

12 

49 

25 

49 

12 

49 

21 

49 

10 

60 

20 

60 

22,903,000 

22,867,000 

33,681,000 

30,461,000 

38,306,000 

35,035,000 

53,807,000 

52,738,000 

11,690,000 

11,758,000 

SELECTED 

^ _  PUN 


PERCENT 

LEVEL 

PAX- 

C57 

PERCENT 

JJ-VEL 

TRADE* 

OFF 

17 

2.8 

45 

3.4 

34 

4.S 

38 

4.6 

19 

8.2 

45 

8.2 

14 

5.4 

57 

S.4 

18 

6,  S 

16 

6.9 

18 

1.3 

IS 

1.3 

30 

2.6 

28 

2.6 

20 

3.2 

20 

3.2 

29 

3.3 

28 

3.3 

20 

3.4 

19 

3.3 

33 

2.9 

35 

2.9 

65 

7.S 

66 

2.5 

151 

2.5 

162 

2.5 

68 

2.5 

70 

2.5 

16 

C.9 

17 

1.7 

37 

67 

15 

67 

42 

49 

16 

49 

38 

49 

16 

49 

35 

49 

20 

49 

30 

70 

19 

60 

22,893,000 

22,908,000 

34,107,000 

34,219,000 

35,026,000 

38,407,1 

D00 

53,219,000 

S3, 926, < 

900 

11,912,000 

11,638,000 
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footnotes 


The  generic  term  "deci sion-maker"  connotes  here  one  or  more  decision-makers 
(see  section  3).  This  review  of  the  SWT  method  is  based  on  the  work  by  Haimes 
and  Hall  (1974).  Haimes,  Hall  and  Freedman  (1975),  Hall  and  Haimes  (1976),  and 
Haimes  (1977). 


2.  See  Haimes  and  Hall  (1974). 

3  The  e-constraint  approach  is  discussed  by  Haimes  (1970),  Haimes,  Lasdon  and 
Wismer  (1971),  and  Haimes  (1975). 

4,  See  Luenberger  (1973)  and  Haimes  (1977). 

5.  See  Everett  (1963)  and  Haimes,  Hall,  and  Freedman  (1975). 

6  See  Haimes,  Hall,  and  Freedman  (1975).  In  addition,  several  variational 
*  approaches  for  the  determination  of  X.j(-)  are  discussed  in  the  above  book. 

7.  See,  for  example,  Raiffa  (1968),  Marschak  (1955),  Fishburn  (1970),  etc. 

E.  See  Kaplan  (1975),  Passy  and  Haimes  (1975),  and  Haimes  (1977). 

9.  See  Haimes,  Hall,  and  Freedman  (1975). 


\Z.  See,  for  example,  Luenberger  (1973). 


1*,.  See  Everett  (1963). 

13.  This  section  is  based  on  the  paper  by  Haimes  and  Chankong  (1979).  The  reader 
is  also  referred  to  Chankong  (1977). 

13.  See  Luenberger  (1973),  page  236. 

K.  Proof  of  these  corollaries  can  be  found  in  Chankong  (1977),  and  Haimes  and 
Chankong  (1979). 

1:.  A  detailed  discussion  on  the  SWT  method  with  multiple  decision-makers  can  be 
found  in  Hall  and  Haimes  (1976),  and  Haimes  (1977). 

16.  See  Hall  and  Haimes  (1976),  and  Haimes  (1977). 

T.  See  Haimes  (1978),  and  Haimes,  Loparo,  Olenik,  and  Nanda  (1979). 

12.  See  Haimes,  Hall,  and  Freedman  (1975),  Haimes  and  Hall  (1977),  and  Haimes  (1977). 

13.  See  Haimes  and  Hall  (1977)  and  Olenik  and  Haimes  (1979). 

23.  See  Haimes  and  Hall  (1977). 

21.  See  Haimes  and  Hall  (1977). 


23.  See,  for  example,  Haimes,  Das  and  Sung  (1977,1979),  Haimes  (1977),  and  Das  and 
Haimes  (1979). 


22.  See,  for  example,  Haimes  (1977). 

2-.  See  Hall  and  Haimes  (1976). 

22.  See  Cohen  and  Marks  (1975). 

Ii.  See  Haines,  Hall,  and  Freedman  (1975). 


27. 


See  Chankong  (1977),  and  Chankong  and  Haimes  (1978). 
28.  See  Haimes,  Loparo,  Olenik,  and  Nanda  (1979). 
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